sc.textFile("/home/brecht-d-m/map/input.nt")) works perfectly.įile permissions for specific file is set to R+W for all users. Trying this in Spark, gives the IOException that seek is not supported ( ). Specifying the whole path (/home/brecht-d-m/map/input.nt) also does not work (as expected, since this also does not work in curl "server denied you to change to the given directory").
#Best ftp client spark code#
Is there an error in the code snippet I gave above, or is that code totally wrong?
I do not see what I am doing wrong in the Scala code. +- TungstenAggregate(key=, functions=, output=)Ĭaused by: .InvalidInputException: Input path does not exist: would assume that the file would be unreachable, but this is in contradiction with that I am able to retrieve the file via curl: curl will print out the specific file on my terminal. +- TungstenExchange SinglePartition, None TungstenAggregate(key=, functions=, output=) Using curl, I am able to download the file, so the path I uses exists.īelow is a snippet of the code I try to execute: var file = sc.textFile("ftp://user:pwd/192.168.1.5/brecht-d-m/map/input.nt")Īfter trying to execute a count on the dataframe, I get following stacktrace ( ): .$TreeNodeException: execute, tree:
Currently, I have followed an example in the Learning Spark repo of Databricks on GitHub. I am trying to read a file on an remote machine in Apache Spark (the Scala version) using ftp.
0 kommentar(er)
