KB 2803: RECOVER THE NUMBER OF USERS PASSING THROUGH THE OLFEO PROXY
Get information about your proxy usage, such as the number of users over the last 30 days and their names.
BACKGROUND
It is not possible to obtain the number of users who have passed through the Olfeo proxy over a given period directly in the Olfeo Statistics section. This is because a limit of 300 entities has been set for performance reasons.
However, this information can be obtained simply by querying the Elasticsearch database that stores olfeo's browsing logs.
STEPS
To obtain this information, please connect via SSH to the Olfeo proxy (the master if you have a cluster).
Once logged in, if you are using Olfeo version 6, enter the chrooted environment with the command: chroot /opt/olfeo/chroot
If you are using version 7, this step is not necessary.
Get the number of unique users over 30 days
You can then run the query in Elasticsearch, which will return the number of unique users over 30 days present in the database:
curl -d '{"size":0,"aggs":{"primary":{"terms":{"field":"user","size":1000000},"aggs":{"count":{"sum":{"field":"hits"}}}}},"query":{"bool":{"must":[{"range":{"timerange":{"lte":"now/d","gte":"now/d-30d"}}}]}}}' -H "Content-Type: application/json" -X POST "http://127.0.0.1:9200/url_*/_search" | python3 -c 'import json,sys;print("Nombre utilisateurs: ", len(json.load(sys.stdin)["aggregations"]["primary"]["buckets"]))'
Get the list of users
To obtain a detailed list of users, run the following command:
curl -H "Content-Type: application/json" -d '{"aggs":{"names": {"terms": {"field": "name.keyword", "size":"5000"}}}, "query":{"bool":{"must":[{"range":{"timerange":{"lte":"now/d","gte":"now/d-30d"}}}]}}}' 'http://127.0.0.1:9200/url_*/_search?pretty=true&size=0' | python3 -c 'import json,sys;[print(e["key"]) for e in json.load(sys.stdin)["aggregations"]["names"]["buckets"]]'
Redirect the user list to a file
If you want to redirect this list of users to a file:
curl -H "Content-Type: application/json" -d '{"aggs":{"names": {"terms": {"field": "name.keyword", "size":"5000"}}}, "query":{"bool":{"must":[{"range":{"timerange":{"lte":"now/d","gte":"now/d-30d"}}}]}}}' 'http://127.0.0.1:9200/url_*/_search?pretty=true&size=0' | python3 -c 'import json,sys;[print(e["key"]) for e in json.load(sys.stdin)["aggregations"]["names"]["buckets"]]' > /tmp/liste_utilisateurs.txt
Interactively inspect the list of users
Finally, to interactively inspect this list, type:
curl -H "Content-Type: application/json" -d '{"aggs":{"names": {"terms": {"field": "name.keyword", "size":"5000"}}}, "query":{"bool":{"must":[{"range":{"timerange":{"lte":"now/d","gte":"now/d-30d"}}}]}}}' 'http://127.0.0.1:9200/url_*/_search?pretty=true&size=0' | python3 -c 'import json,sys;[print(e["key"]) for e in json.load(sys.stdin)["aggregations"]["names"]["buckets"]]' | less
VALIDATION
You can then obtain more detailed information (domains visited, browsing volume) about a given user via the olfeo Webadmin statistics engine.