 |
simdjson
0.7.0
Ridiculously Fast JSON
|
|
simdjson
0.7.0
Ridiculously Fast JSON
|
An overview of what you need to know to use simdjson, with examples.
To include simdjson, copy the simdjson.h and simdjson.cpp files from the singleheader directory into your project. Then include the header file in your project with:
You can compile with:
Note:
c++ -std=c++17 myproject.cpp simdjson.cpp)._CRT_SECURE_NO_WARNINGS flag to avoid warnings with respect to our use of standard C functions such as fopen.You can install the simdjson library on your system or in your project using multiple package managers such as MSYS2, the conan package manager, vcpkg, brew, the apt package manager (debian-based Linux systems), the FreeBSD package manager (FreeBSD), and so on. Visit our wiki for more details.
You can include the simdjson as a CMake dependency by including the following lines in your CMakeLists.txt:
You should replace GIT_TAG v0.6.1 by the version you need. If you omit GIT_TAG v0.6.1, you will work from the main branch of simdjson: we recommend that if you are working on production code,
Elsewhere in your project, you can declare dependencies on simdjson with lines such as these:
We recommend CMake version 3.15 or better.
See our CMake demonstration. It works under Linux, FreeBSD, macOS and Windows (including Visual Studio).
The CMake build in simdjson can be taylored with a few variables. You can see the available variables and their default values by entering the cmake -LA command.
The simdjson library offers a simple DOM tree API, which you can access by creating a dom::parser and calling the load() method:
Or by creating a padded string (for efficiency reasons, simdjson requires a string with SIMDJSON_PADDING bytes at the end) and calling parse():
The parsed document resulting from the parser.load and parser.parse calls depends on the parser instance. Thus the parser instance must remain in scope. Furthermore, you must have at most one parsed document in play per parser instance. You cannot copy a parser instance, you may only move it.
If you need to keep a document around long term, you can keep or move the parser instance. Note that moving a parser instance, or keeping one in a movable data structure like vector or map, can cause any outstanding element, object or array instances to be invalidated. If you need to store a parser in a movable data structure, you should use a std::unique_ptr to avoid this invalidation(e.g., std::unique_ptr<dom::parser> parser(new dom::parser{})).
During theload or parse calls, neither the input file nor the input string are ever modified. After calling load or parse, the source (either a file or a string) can be safely discarded. All of the JSON data is stored in the parser instance. The parsed document is also immutable in simdjson: you do not modify it by accessing it.
For best performance, a parser instance should be reused over several files: otherwise you will needlessly reallocate memory, an expensive process. It is also possible to avoid entirely memory allocations during parsing when using simdjson.
If you need a lower-level interface, you may call the function parser.parse(const char * p, size_t l) on a pointer p while specifying the length of your input l in bytes. To see how to get the very best performance from a low-level approach, you way want to read our performance notes on this topic (see the Padding and Temporary Copies section).
Once you have an element, you can navigate it with idiomatic C++ iterators, operators and casts.
double(element) or double x = json_element. This works for double, uint64_t, int64_t, bool, dom::object and dom::array. An exception is thrown if the cast is not possible.get() with error codes to avoid exceptions. You first declare the variable of the appropriate type (double, uint64_t, int64_t, bool, dom::object and dom::array) and pass it by reference to get() which gives you back an error code: e.g., object["foo"].for (auto value : array) { ... }. If you know the type of the value, you can cast it right there, too! for (double value : array) { ... }for (auto [key, value] : object)array.at(0) gets the first element. Note that array[0] does not compile, because implementing [] gives the impression indexing is a O(1) operation, which it is not presently in simdjson. Instead, you should iterate over the elements using a for-loop, as in our examples.
size() method.element.type(). It returns an element_type with values such as simdjson::dom::element_type::ARRAY, simdjson::dom::element_type::OBJECT, simdjson::dom::element_type::INT64, simdjson::dom::element_type::UINT64,simdjson::dom::element_type::DOUBLE, simdjson::dom::element_type::BOOL or, simdjson::dom::element_type::NULL_VALUE.out << element). You can also request the construction of a minified string version (simdjson::minify(element)).The following code illustrates all of the above:
Here is a different example illustrating the same ideas:
And another one:
The simdjson library builds on compilers supporting the C++11 standard. It is also a strict requirement: we have no plan to support older C++ compilers.
We represent parsed strings in simdjson using the std::string_view class. It avoids the need to copy the data, as would be necessary with the std::string class. It also avoids the pitfalls of null-terminated C strings.
The std::string_view class has become standard as part of C++17 but it is not always available on compilers which only supports C++11. When we detect that string_view is natively available, we define the macro SIMDJSON_HAS_STRING_VIEW.
When we detect that it is unavailable, we use string-view-lite as a substitute. In such cases, we use the type alias using string_view = nonstd::string_view; to offer the same API, irrespective of the compiler and standard library. The macro SIMDJSON_HAS_STRING_VIEW will be undefined to indicate that we emulate string_view.
While the simdjson library can be used in any project using C++ 11 and above, field iteration has special support C++ 17's destructuring syntax. For example:
For comparison, here is the C++ 11 version of the same code:
In some cases, you may have valid JSON strings that you do not wish to parse but that you wish to minify. That is, you wish to remove all unnecessary spaces. We have a fast function for this purpose (simdjson::minify(const char * input, size_t length, const char * output, size_t& new_length)). This function does not validate your content, and it does not parse it. It is much faster than parsing the string and re-serializing it in minified form (simdjson::minify(parser.parse())). Usage is relatively simple. You must pass an input pointer with a length parameter, as well as an output pointer and an output length parameter (by reference). The output length parameter is not read, but written to. The output pointer should point to a valid memory region that is as large as the original string length. The input pointer and input length are read, but not written to.
Though it does not validate the JSON input, it will detect when the document ends with an unterminated string. E.g., it would refuse to minify the string "this string is not terminated because of the missing final quote.
The simdjson library has fast functions to validate UTF-8 strings. They are many times faster than most functions commonly found in libraries. You can use our fast functions, even if you do not care about JSON.
The UTF-8 validation function merely checks that the input is valid UTF-8: it works with strings in general, not just JSON strings.
Your input string does not need any padding. Any string will do. The validate_utf8 function does not do any memory allocation on the heap, and it does not throw exceptions.
The simdjson library also supports JSON pointer through the at_pointer() method, letting you reach further down into the document in a single call:
A JSON Path is a sequence of segments each starting with the '/' character. Within arrays, an integer index allows you to select the indexed node. Within objects, the string value of the key allows you to select the value. If your keys contain the characters '/' or '~', they must be escaped as '~1' and '~0' respectively. An empty JSON Path refers to the whole document.
We also extend the JSON Pointer support to include relative paths. You can apply a JSON path to any node and the path gets interpreted relatively, as if the currrent node were a whole JSON document.
Consider the following example:
All simdjson APIs that can fail return simdjson_result<T>, which is a <value, error_code> pair. You can retrieve the value with .get(), like so:
When you use the code this way, it is your responsibility to check for error before using the result: if there is an error, the result value will not be valid and using it will caused undefined behavior.
We can write a "quick start" example where we attempt to parse the following JSON file and access some data, without triggering exceptions:
Our program loads the file, selects value corresponding to key "search_metadata" which expected to be an object, and then it selects the key "count" within that object.
The following is a similar example where one wants to get the id of the first tweet without triggering exceptions. To do this, we use ["statuses"].at(0)["id"]. We break that expression down:
"statuses" key of the document) using ["statuses"]). The result is expected to be an array..at(0). The result is expected to be an object.Observe how we use the at method when querying an index into an array, and not the bracket operator.
This is how the example in "Using the Parsed JSON" could be written using only error code checking:
Here is another example:
And another one:
Notice how we can string several operations (parser.parse(abstract_json)["str"]["123"]["abc"].get(v)) and only check for the error once, a strategy we call error chaining.
The next two functions will take as input a JSON document containing an array with a single element, either a string or a number. They return true upon success.
Users more comfortable with an exception flow may choose to directly cast the simdjson_result<T> to the desired type:
When used this way, a simdjson_error exception will be thrown if an error occurs, preventing the program from continuing if there was an error.
If one is willing to trigger exceptions, it is possible to write simpler code:
Sometimes you don't necessarily have a document with a known type, and are trying to generically inspect or walk over JSON elements. To do that, you can use iterators and the type() method. For example, here's a quick and dirty recursive function that verbosely prints the JSON document as JSON (* ignoring nuances like trailing commas and escaping strings, for brevity's sake):
The simdjson library also support multithreaded JSON streaming through a large file containing many smaller JSON documents in either ndjson or JSON lines format. If your JSON documents all contain arrays or objects, we even support direct file concatenation without whitespace. The concatenated file has no size restrictions (including larger than 4GB), though each individual document must be no larger than 4 GB.
Here is a simple example, given "x.json" with this content:
In-memory ndjson strings can be parsed as well, with parser.parse_many(string):
Unlike parser.parse, both parser.load_many(filename) and parser.parse_many(string) may parse "on demand" (lazily). That is, no parsing may have been done before you enter the loop for (dom::element doc : docs) { and you should expect the parser to only ever fully parse one JSON document at a time.
parser.load_many(filename), the file's content is loaded up in a memory buffer owned by the parser's instance. Thus the file can be safely deleted after calling parser.load_many(filename) as the parser instance owns all of the data.parser.parse_many(string), no copy is made of the provided string input. The provided memory buffer may be accessed each time a JSON document is parsed. Calling parser.parse_many(string) on a temporary string buffer (e.g., docs = parser.parse_many("[1,2,3]"_padded)) is unsafe (and will not compile) because the document_stream instance needs access to the buffer to return the JSON documents. In constrast, calling doc = parser.parse("[1,2,3]"_padded) is safe because parser.parse eagerly parses the input.Both load_many and parse_many take an optional parameter size_t batch_size which defines the window processing size. It is set by default to a large value (1000000 corresponding to 1 MB). None of your JSON documents should exceed this window size, or else you will get the error simdjson::CAPACITY. You cannot set this window size larger than 4 GB: you will get the error simdjson::CAPACITY. The smaller the window size is, the less memory the function will use. Setting the window size too small (e.g., less than 100 kB) may also impact performance negatively. Leaving it to 1 MB is expected to be a good choice, unless you have some larger documents.
We built simdjson with thread safety in mind.
The simdjson library is single-threaded except for parse_many which may use secondary threads under its control when the library is compiled with thread support.
We recommend using one dom::parser object per thread in which case the library is thread-safe. It is unsafe to reuse a dom::parser object between different threads. The parsed results (dom::document, dom::element, array, object) depend on the dom::parser, etc. therefore it is also potentially unsafe to use the result of the parsing between different threads.
The CPU detection, which runs the first time parsing is attempted and switches to the fastest parser for your CPU, is transparent and thread-safe.
The only header file supported by simdjson is simdjson.h. Older versions of simdjson published a number of other include files such as document.h or ParsedJson.h alongside simdjson.h; these headers may be moved or removed in future versions.
1.8.20