Viz Social User Guide

Version 1.1 | Published January 26, 2022 ©

Volume and Performance Limits

The performance and throughput of a standard Viz Social server is limited by a variety of parameters. These are listed for a standard Viz Social server and where possible an indication is given when a limit can be lifted by means of a server upgrade.

The performance and throughput of a standard Viz Social server is limited by a variety of parameters. In this Appendix, these are listed for a standard Viz Social server and where possible an indication is given when a limit can be lifted by means of a server upgrade.

Global Limits

Here are some key figures applicable to standard Viz Social installations.

  • The maximum number of Searches on a Viz Social server is 100.

  • Standard resources suffice for storing 3M posts per Search and 10M posts in total. When more messages are stored the risk increases that part of the data can no longer be indexed in memory, which will eventually dramatically slow down system response times.

    Note: This threshold can be increased by a (temporary) upgrade of computer resources.

  • Once the appropriate Searches have been set-up, Viz Social’s data collection runs as a continuous and autonomous background process, if necessary for multiple productions in parallel. For the curation and publication of feeds with more than 100k posts, we recommend working on only one production at a time, with at most three users logged on, and preferably only one user with the final Go/No-go authority (to prevent decision clashes and/or misunderstandings).

Format Limits

Aggregated Formats continuously consume system resources and thus may affect system performance. Hence, we advise adhering to the following guidelines:

  • For large collections, i.e. more than 100k posts, we recommend running at most five unrestricted or three restricted Polls simultaneously on one server.

    Note: For a restricted Poll the system limits the number of votes for each user and counts only the first or last N ones. For an unrestricted Poll, no limit is imposed.

  • There is no limit on the number of Competitions, but the draw of a winner from 1M+ posts might take longer than ten seconds (depending on the chosen business logic of the competition).

Twitter

Search API Limits (“Low Volume, incl. History” Searches)

The integration of Twitter’s Search or REST API into Viz Social is described in Twitter. Known volume or performance restrictions are:

  • Twitter limits API access when more than 450 calls per interval of 15 minutes are made. With the default Viz Social setup, where Searches are repeated once a minute, this implies that the interface will be rate-limited by Twitter when more than 30 (=450/15) of these Searches are active on the platform at the same time. The effect of rate limiting is that Searches return no results until the 15-minute interval is over.

  • Rate limiting kicks in even earlier for Searches that return a lot of hits, typically more than 100 hits/minute. Each page of 100 results after the first one requires a new request and thus effectively counts as a new Search.

  • In reality, the effective maximum throughput strongly depends on how efficient the pages are filled. If all of them are full the theoretical maximum throughput is 50 Tweets/sec. If each of them just contains a single Tweet, then the effective throughput is only 0.5 Tweets/sec. In reality, the Twitter API rarely uses all available bandwidth, so +/- 5 Tweets/sec is more a typical number often observes as a maximum for this API.

  • Each Search phrase individually must be less than 500 characters.

  • At most 20 Twitter users can be monitored per Search.

The maximum holds per server for the combined volume of all active Twitter Searches using the Twitter Search API. In case these limits are too restrictive, we advise using Twitter’s Streaming API instead.

Streaming API Limits (“High Volume, no History” Searches)

The integration of Twitter’s Streaming API into Viz Social is described in Twitter. Known volume or performance restrictions are:

  • Twitter limits the returned volume of Tweets over its Streaming API to roughly 1% of the momentary aggregated worldwide volume. This roughly translates into an average API maximum of 60 Tweets/sec on a normal day. During peaked, special events like the Oscars or the World Cup Finale, this may well increase to over a 100 Tweets/sec or more.

  • Viz Social is able to harvest at a rate of 15-20 messages/sec from this API.

  • The number of Search creations, modifications, deletions should be less than one per minute to prevent Twitter from temporarily suspending traffic.

  • At any point in time, the total number of searched terms on the installation should not exceed 400 and the total number of monitored users on the installation should not exceed 5000.

The maximum holds per server for the combined volume of all active Searches using the Twitter Streaming API.

Filtered Firehose Limits

The integration of Twitter’s Firehose API into Viz Social is described in Twitter. It is guaranteed complete, independent of volume and rate.

Nevertheless, each installation with finite physical resources (CPU, memory, …) necessarily has its limits. When Tweets start to arrive at a faster rate than the CPU can process, the surplus will be queued up and, as a result, users will start noticing delays. This continues until the max queue size (dictated by the size of the system’s memory) is reached, after which the surplus is ignored because there’s simply no place anymore for them.

Hence, it is crucial that the installed hardware is in line with the expected traffic profile when every Tweet matters. To give some guidance:

  • A standard Viz Social server can process 200 Tweets/sec without queues forming*.

  • When traffic levels are further increased to 1000 Tweets/sec, a standard virgin Viz Social server can withhold and queue up messages for at least 15 minutes*.

  • These numbers are affected adversely by:

    • Active Formats, particularly restricted Polls.

    • Different users logged in and active on the same Format/Search.

    • Many messages already stored on the platform (3-5M or more).

  • Capacity figures can be further increased by scaling horizontally and/or vertically.

Auto Reply Limits

Twitter restricts the number of Tweets originating from any account to 2400 per day (imposed in 30 minute intervals) and this limit also applies to replies automatically generated by Viz Social. For certain cases, always explicitly agreed in advance, Twitter may be able to raise the limit well-beyond its default value. Support is able to address this matter via their Twitter contacts, so please contact your representative in case the standard rate is insufficient for your use cases.

Instagram

API Limits

The integration of Instagram into Viz Social is described in Instagram. Instagram’s API allows Viz Social to run 83 Instagram Searches in parallel per server.

  • Instagram limits API access when more than 5000 calls per hour are made. With the default Viz Social set-up where Searches are repeated once a minute, this implies that the interface will be rate-limited by Instagram when more than 83 (=5000/60) of these Searches are active on the platform at the same time. The effect of rate limiting will be that Searches will return no results until the hour interval is over.

  • Rate limiting kicks in even earlier for Searches that return a lot of hits, typically more than 25 hits/minute. Each page of 25 results after the first one requires a new request and thus effectively counts as a new Search.

  • In reality, the effective maximum throughput strongly depends on how efficient the pages are filled. If all of them are full the theoretical maximum throughput is 35 Posts/sec. If each of them just contains a single post, then the effective throughput is only 1.4 Posts/sec. In reality, the Instagram API rarely uses all available bandwidth, so +/- 3-5 Posts/sec is a more typical number often observes as a maximum for this API.

Facebook

Graph API Limits (Page and Post Searches)

The integration of the Facebook Graph API into Viz Social forms the basis of the Facebook Page and Post Searches and is described in Facebook. Known volume or performance restrictions are:

  • Facebook limits API access when more than 600 searches/calls per interval of ten minutes are made. With the default Viz Social set-up where Searches are repeated once a minute, this implies that the interface will be rate-limited by Facebook when more than 60 (=600/10) Facebook Page, Term & Post Searches instances are active at the same time. The effect of rate limiting will be that Searches will return no results until the 10-minute interval is over.

  • Rate limiting kicks in even earlier for Searches that return a lot of hits, typically more than ten hits/minute. Each page of ten results after the first one effectively counts as a new Search.

  • When a Facebook Page Search also monitors the comments of N posts, this counts as N Searches.

  • In reality the effective maximum throughput strongly depends on how efficient the pages are filled. If all of them are full the theoretical maximum throughput is 10 Posts/sec. If each of them just contains a single post, then the effective throughput is only 1.0 Posts/sec. In reality the Instagram API rarely uses all available bandwidth, so +/- 3 Posts/sec is a more typical number often observes as a maximum for this API.

Social Reaction Limits

Viz Social’s social reactions use the public APIs of the various social networks, through which certain limits are imposed on the user’s behavior.

Twitter:

  • At most 900 Fetches and Updates combined per 900 seconds.

  • At most 15 Likes per 15 minutes.

  • At most 15 Retweets per 15 minutes

  • At most 15 Replies and Tweets combined per 15 minutes.

Instagram:

  • No limit on the number of Updates.

  • At most 60 Comments per 60 minutes.

Facebook:

  • Officially no limit, but in practice at most 600 Updates, Likes, Shares, Comments and Posts combined per 600 seconds.

Note: Exceeding these limits might result in a temporary ban by the social network (usually between 10-30 minutes, but up to the discretion of the network). The ban may last longer or even become permanent in case limits are crossed consistently and repetitively, particularly when occurring within a short period of time.