![]() # "$LD_LIBRARY_PATH:/usr/lib/R/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/default-java/lib/server" Here is an example of initiating the spark session while also setting the configuration options: You can specify a different codec, such as "gzip", during the session initialization. .codec: by default, spark compresses parquet files by snappy codec which creates rather large temporary files.If this is too small, you get java out-of-memory errors related to java heap space. Use character strings, such as "500m" or "4g". : how much memory to allocate to the spark driver (or the complete spark process if run locally).sparkConfig is a list of configuration options. ![]() ![]() You can also use "local" if you want to utilize all logical cpus (one process per cpu thread). ![]() "local" mean spark will be run locally, on your computer, utilizing n cpus. master is the location the spark master (driver). ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |