Package org.apache.sysds.api.mlcontext
Class MLContextConversionUtil
- java.lang.Object
-
- org.apache.sysds.api.mlcontext.MLContextConversionUtil
-
public class MLContextConversionUtil extends Object
Utility class containing methods to perform data conversions.
-
-
Constructor Summary
Constructors Constructor Description MLContextConversionUtil()
-
Method Summary
All Methods Static Methods Concrete Methods Modifier and Type Method Description static FrameObjectbinaryBlocksToFrameObject(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> binaryBlocks)Convert aJavaPairRDD<Long, FrameBlock>to aFrameObject.static FrameObjectbinaryBlocksToFrameObject(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> binaryBlocks, FrameMetadata frameMetadata)Convert aJavaPairRDD<Long, FrameBlock>to aFrameObject.static MatrixBlockbinaryBlocksToMatrixBlock(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> binaryBlocks, MatrixMetadata matrixMetadata)Convert aJavaPairRDD<MatrixIndexes, MatrixBlock>to aMatrixBlockstatic MatrixObjectbinaryBlocksToMatrixObject(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> binaryBlocks)Convert aJavaPairRDD<MatrixIndexes, MatrixBlock>to aMatrixObject.static MatrixObjectbinaryBlocksToMatrixObject(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> binaryBlocks, MatrixMetadata matrixMetadata)Convert aJavaPairRDD<MatrixIndexes, MatrixBlock>to aMatrixObject.static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock>dataFrameToFrameBinaryBlocks(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame, FrameMetadata frameMetadata)Convert aDataFrameto aJavaPairRDD<Long, FrameBlock>binary-block frame.static FrameObjectdataFrameToFrameObject(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame)Convert aDataFrameto aFrameObject.static FrameObjectdataFrameToFrameObject(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame, FrameMetadata frameMetadata)Convert aDataFrameto aFrameObject.static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock>dataFrameToMatrixBinaryBlocks(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame)Convert aDataFrameto aJavaPairRDD<MatrixIndexes, MatrixBlock>binary-block matrix.static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock>dataFrameToMatrixBinaryBlocks(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame, MatrixMetadata matrixMetadata)Convert aDataFrameto aJavaPairRDD<MatrixIndexes, MatrixBlock>binary-block matrix.static MatrixObjectdataFrameToMatrixObject(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame)Convert aDataFrameto aMatrixObject.static MatrixObjectdataFrameToMatrixObject(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame, MatrixMetadata matrixMetadata)Convert aDataFrameto aMatrixObject.static voiddetermineFrameFormatIfNeeded(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame, FrameMetadata frameMetadata)If the FrameFormat of the DataFrame has not been explicitly specified, attempt to determine the proper FrameFormat.static voiddetermineMatrixFormatIfNeeded(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame, MatrixMetadata matrixMetadata)If the MatrixFormat of the DataFrame has not been explicitly specified, attempt to determine the proper MatrixFormat.static MatrixObjectdoubleMatrixToMatrixObject(String variableName, double[][] doubleMatrix)Convert a two-dimensional double array to aMatrixObject.static MatrixObjectdoubleMatrixToMatrixObject(String variableName, double[][] doubleMatrix, MatrixMetadata matrixMetadata)Convert a two-dimensional double array to aMatrixObject.static FrameObjectframeBlockToFrameObject(String variableName, FrameBlock frameBlock, FrameMetadata frameMetadata)Convert aFrameBlockto aFrameObject.static String[][]frameObjectTo2DStringArray(FrameObject frameObject)Convert aFrameObjectto a two-dimensional string array.static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock>frameObjectToBinaryBlocks(FrameObject frameObject, SparkExecutionContext sparkExecutionContext)Convert aFrameObjectto aJavaPairRDD<Long, FrameBlock>.static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>frameObjectToDataFrame(FrameObject frameObject, SparkExecutionContext sparkExecutionContext)Convert aFrameObjectto aDataFrame.static org.apache.spark.api.java.JavaRDD<String>frameObjectToJavaRDDStringCSV(FrameObject frameObject, String delimiter)Convert aFrameObjectto aJavaRDD<String>in CSV format.static org.apache.spark.api.java.JavaRDD<String>frameObjectToJavaRDDStringIJV(FrameObject frameObject)Convert aFrameObjectto aJavaRDD<String>in IJV format.static List<String>frameObjectToListStringCSV(FrameObject frameObject, String delimiter)Convert aFrameObjectto aList<String>in CSV format.static List<String>frameObjectToListStringIJV(FrameObject frameObject)Convert aFrameObjectto aList<String>in IJV format.static org.apache.spark.rdd.RDD<String>frameObjectToRDDStringCSV(FrameObject frameObject, String delimiter)Convert aFrameObjectto aRDD<String>in CSV format.static org.apache.spark.rdd.RDD<String>frameObjectToRDDStringIJV(FrameObject frameObject)Convert aFrameObjectto aRDD<String>in IJV format.static booleanisDataFrameWithIDColumn(FrameMetadata frameMetadata)Return whether or not the DataFrame has an ID column.static booleanisDataFrameWithIDColumn(MatrixMetadata matrixMetadata)Return whether or not the DataFrame has an ID column.static booleanisVectorBasedDataFrame(MatrixMetadata matrixMetadata)Return whether or not the DataFrame is vector-based.static FrameObjectjavaRDDStringCSVToFrameObject(org.apache.spark.api.java.JavaRDD<String> javaRDD)Convert aJavaRDD<String>in CSV format to aFrameObjectstatic FrameObjectjavaRDDStringCSVToFrameObject(org.apache.spark.api.java.JavaRDD<String> javaRDD, FrameMetadata frameMetadata)Convert aJavaRDD<String>in CSV format to aFrameObjectstatic MatrixObjectjavaRDDStringCSVToMatrixObject(org.apache.spark.api.java.JavaRDD<String> javaRDD)Convert aJavaRDD<String>in CSV format to aMatrixObjectstatic MatrixObjectjavaRDDStringCSVToMatrixObject(org.apache.spark.api.java.JavaRDD<String> javaRDD, MatrixMetadata matrixMetadata)Convert aJavaRDD<String>in CSV format to aMatrixObjectstatic FrameObjectjavaRDDStringIJVToFrameObject(org.apache.spark.api.java.JavaRDD<String> javaRDD, FrameMetadata frameMetadata)Convert aJavaRDD<String>in IJV format to aFrameObject.static MatrixObjectjavaRDDStringIJVToMatrixObject(org.apache.spark.api.java.JavaRDD<String> javaRDD, MatrixMetadata matrixMetadata)Convert aJavaRDD<String>in IJV format to aMatrixObject.static org.apache.spark.api.java.JavaSparkContextjsc()Obtain JavaSparkContext from MLContextProxy.static MatrixObjectmatrixBlockToMatrixObject(String variableName, MatrixBlock matrixBlock, MatrixMetadata matrixMetadata)Convert aMatrixBlockto aMatrixObject.static double[][]matrixObjectTo2DDoubleArray(MatrixObject matrixObject)Convert aMatrixObjectto a two-dimensional double array.static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock>matrixObjectToBinaryBlocks(MatrixObject matrixObject, SparkExecutionContext sparkExecutionContext)Convert aMatrixObjectto aJavaPairRDD<MatrixIndexes, MatrixBlock>.static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row>matrixObjectToDataFrame(MatrixObject matrixObject, SparkExecutionContext sparkExecutionContext, boolean isVectorDF)Convert aMatrixObjectto aDataFrame.static org.apache.spark.api.java.JavaRDD<String>matrixObjectToJavaRDDStringCSV(MatrixObject matrixObject)Convert aMatrixObjectto aJavaRDD<String>in CSV format.static org.apache.spark.api.java.JavaRDD<String>matrixObjectToJavaRDDStringIJV(MatrixObject matrixObject)Convert aMatrixObjectto aJavaRDD<String>in IJV format.static List<String>matrixObjectToListStringCSV(MatrixObject matrixObject)Convert aMatrixObjectto aList<String>in CSV format.static List<String>matrixObjectToListStringIJV(MatrixObject matrixObject)Convert aMatrixObjectto aList<String>in IJV format.static org.apache.spark.rdd.RDD<String>matrixObjectToRDDStringCSV(MatrixObject matrixObject)Convert aMatrixObjectto aRDD<String>in CSV format.static org.apache.spark.rdd.RDD<String>matrixObjectToRDDStringIJV(MatrixObject matrixObject)Convert aMatrixObjectto aRDD<String>in IJV format.static FrameObjectrddStringCSVToFrameObject(org.apache.spark.rdd.RDD<String> rdd)Convert aRDD<String>in CSV format to aFrameObjectstatic FrameObjectrddStringCSVToFrameObject(org.apache.spark.rdd.RDD<String> rdd, FrameMetadata frameMetadata)Convert aRDD<String>in CSV format to aFrameObjectstatic MatrixObjectrddStringCSVToMatrixObject(org.apache.spark.rdd.RDD<String> rdd)Convert aRDD<String>in CSV format to aMatrixObjectstatic MatrixObjectrddStringCSVToMatrixObject(org.apache.spark.rdd.RDD<String> rdd, MatrixMetadata matrixMetadata)Convert aRDD<String>in CSV format to aMatrixObjectstatic FrameObjectrddStringIJVToFrameObject(org.apache.spark.rdd.RDD<String> rdd, FrameMetadata frameMetadata)Convert aRDD<String>in IJV format to aFrameObject.static MatrixObjectrddStringIJVToMatrixObject(org.apache.spark.rdd.RDD<String> rdd, MatrixMetadata matrixMetadata)Convert aRDD<String>in IJV format to aMatrixObject.static org.apache.spark.SparkContextsc()Obtain SparkContext from MLContextProxy.static org.apache.spark.sql.SparkSessionspark()Obtain SparkSession from MLContextProxy.static MatrixObjecturlToMatrixObject(URL url, MatrixMetadata matrixMetadata)Convert a matrix at a URL to aMatrixObject.
-
-
-
Method Detail
-
doubleMatrixToMatrixObject
public static MatrixObject doubleMatrixToMatrixObject(String variableName, double[][] doubleMatrix)
Convert a two-dimensional double array to aMatrixObject.- Parameters:
variableName- name of the variable associated with the matrixdoubleMatrix- matrix of double values- Returns:
- the two-dimensional double matrix converted to a
MatrixObject
-
doubleMatrixToMatrixObject
public static MatrixObject doubleMatrixToMatrixObject(String variableName, double[][] doubleMatrix, MatrixMetadata matrixMetadata)
Convert a two-dimensional double array to aMatrixObject.- Parameters:
variableName- name of the variable associated with the matrixdoubleMatrix- matrix of double valuesmatrixMetadata- the matrix metadata- Returns:
- the two-dimensional double matrix converted to a
MatrixObject
-
urlToMatrixObject
public static MatrixObject urlToMatrixObject(URL url, MatrixMetadata matrixMetadata)
Convert a matrix at a URL to aMatrixObject.- Parameters:
url- the URL to a matrix (in CSV or IJV format)matrixMetadata- the matrix metadata- Returns:
- the matrix at a URL converted to a
MatrixObject
-
matrixBlockToMatrixObject
public static MatrixObject matrixBlockToMatrixObject(String variableName, MatrixBlock matrixBlock, MatrixMetadata matrixMetadata)
Convert aMatrixBlockto aMatrixObject.- Parameters:
variableName- name of the variable associated with the matrixmatrixBlock- matrix as a MatrixBlockmatrixMetadata- the matrix metadata- Returns:
- the
MatrixBlockconverted to aMatrixObject
-
frameBlockToFrameObject
public static FrameObject frameBlockToFrameObject(String variableName, FrameBlock frameBlock, FrameMetadata frameMetadata)
Convert aFrameBlockto aFrameObject.- Parameters:
variableName- name of the variable associated with the frameframeBlock- frame as a FrameBlockframeMetadata- the frame metadata- Returns:
- the
FrameBlockconverted to aFrameObject
-
binaryBlocksToMatrixObject
public static MatrixObject binaryBlocksToMatrixObject(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> binaryBlocks)
Convert aJavaPairRDD<MatrixIndexes, MatrixBlock>to aMatrixObject.- Parameters:
binaryBlocks-JavaPairRDD<MatrixIndexes, MatrixBlock>representation of a binary-block matrix- Returns:
- the
JavaPairRDD<MatrixIndexes, MatrixBlock>matrix converted to aMatrixObject
-
binaryBlocksToMatrixObject
public static MatrixObject binaryBlocksToMatrixObject(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> binaryBlocks, MatrixMetadata matrixMetadata)
Convert aJavaPairRDD<MatrixIndexes, MatrixBlock>to aMatrixObject.- Parameters:
binaryBlocks-JavaPairRDD<MatrixIndexes, MatrixBlock>representation of a binary-block matrixmatrixMetadata- the matrix metadata- Returns:
- the
JavaPairRDD<MatrixIndexes, MatrixBlock>matrix converted to aMatrixObject
-
binaryBlocksToMatrixBlock
public static MatrixBlock binaryBlocksToMatrixBlock(org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> binaryBlocks, MatrixMetadata matrixMetadata)
Convert aJavaPairRDD<MatrixIndexes, MatrixBlock>to aMatrixBlock- Parameters:
binaryBlocks-JavaPairRDD<MatrixIndexes, MatrixBlock>representation of a binary-block matrixmatrixMetadata- the matrix metadata- Returns:
- the
JavaPairRDD<MatrixIndexes, MatrixBlock>matrix converted to aMatrixBlock
-
binaryBlocksToFrameObject
public static FrameObject binaryBlocksToFrameObject(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> binaryBlocks)
Convert aJavaPairRDD<Long, FrameBlock>to aFrameObject.- Parameters:
binaryBlocks-JavaPairRDD<Long, FrameBlock>representation of a binary-block frame- Returns:
- the
JavaPairRDD<Long, FrameBlock>frame converted to aFrameObject
-
binaryBlocksToFrameObject
public static FrameObject binaryBlocksToFrameObject(org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> binaryBlocks, FrameMetadata frameMetadata)
Convert aJavaPairRDD<Long, FrameBlock>to aFrameObject.- Parameters:
binaryBlocks-JavaPairRDD<Long, FrameBlock>representation of a binary-block frameframeMetadata- the frame metadata- Returns:
- the
JavaPairRDD<Long, FrameBlock>frame converted to aFrameObject
-
dataFrameToMatrixObject
public static MatrixObject dataFrameToMatrixObject(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame)
Convert aDataFrameto aMatrixObject.- Parameters:
dataFrame- the SparkDataFrame- Returns:
- the
DataFramematrix converted to a converted to aMatrixObject
-
dataFrameToMatrixObject
public static MatrixObject dataFrameToMatrixObject(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame, MatrixMetadata matrixMetadata)
Convert aDataFrameto aMatrixObject.- Parameters:
dataFrame- the SparkDataFramematrixMetadata- the matrix metadata- Returns:
- the
DataFramematrix converted to a converted to aMatrixObject
-
dataFrameToFrameObject
public static FrameObject dataFrameToFrameObject(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame)
Convert aDataFrameto aFrameObject.- Parameters:
dataFrame- the SparkDataFrame- Returns:
- the
DataFramematrix converted to a converted to aFrameObject
-
dataFrameToFrameObject
public static FrameObject dataFrameToFrameObject(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame, FrameMetadata frameMetadata)
Convert aDataFrameto aFrameObject.- Parameters:
dataFrame- the SparkDataFrameframeMetadata- the frame metadata- Returns:
- the
DataFrameframe converted to a converted to aFrameObject
-
dataFrameToMatrixBinaryBlocks
public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> dataFrameToMatrixBinaryBlocks(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame)
Convert aDataFrameto aJavaPairRDD<MatrixIndexes, MatrixBlock>binary-block matrix.- Parameters:
dataFrame- the SparkDataFrame- Returns:
- the
DataFramematrix converted to aJavaPairRDD<MatrixIndexes, MatrixBlock>binary-block matrix
-
dataFrameToMatrixBinaryBlocks
public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> dataFrameToMatrixBinaryBlocks(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame, MatrixMetadata matrixMetadata)
Convert aDataFrameto aJavaPairRDD<MatrixIndexes, MatrixBlock>binary-block matrix.- Parameters:
dataFrame- the SparkDataFramematrixMetadata- the matrix metadata- Returns:
- the
DataFramematrix converted to aJavaPairRDD<MatrixIndexes, MatrixBlock>binary-block matrix
-
dataFrameToFrameBinaryBlocks
public static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> dataFrameToFrameBinaryBlocks(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame, FrameMetadata frameMetadata)
Convert aDataFrameto aJavaPairRDD<Long, FrameBlock>binary-block frame.- Parameters:
dataFrame- the SparkDataFrameframeMetadata- the frame metadata- Returns:
- the
DataFramematrix converted to aJavaPairRDD<Long, FrameBlock>binary-block frame
-
determineMatrixFormatIfNeeded
public static void determineMatrixFormatIfNeeded(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame, MatrixMetadata matrixMetadata)If the MatrixFormat of the DataFrame has not been explicitly specified, attempt to determine the proper MatrixFormat.- Parameters:
dataFrame- the SparkDataFramematrixMetadata- the matrix metadata, if available
-
determineFrameFormatIfNeeded
public static void determineFrameFormatIfNeeded(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> dataFrame, FrameMetadata frameMetadata)If the FrameFormat of the DataFrame has not been explicitly specified, attempt to determine the proper FrameFormat.- Parameters:
dataFrame- the SparkDataFrameframeMetadata- the frame metadata, if available
-
isDataFrameWithIDColumn
public static boolean isDataFrameWithIDColumn(MatrixMetadata matrixMetadata)
Return whether or not the DataFrame has an ID column.- Parameters:
matrixMetadata- the matrix metadata- Returns:
trueif the DataFrame has an ID column,falseotherwise.
-
isDataFrameWithIDColumn
public static boolean isDataFrameWithIDColumn(FrameMetadata frameMetadata)
Return whether or not the DataFrame has an ID column.- Parameters:
frameMetadata- the frame metadata- Returns:
trueif the DataFrame has an ID column,falseotherwise.
-
isVectorBasedDataFrame
public static boolean isVectorBasedDataFrame(MatrixMetadata matrixMetadata)
Return whether or not the DataFrame is vector-based.- Parameters:
matrixMetadata- the matrix metadata- Returns:
trueif the DataFrame is vector-based,falseotherwise.
-
javaRDDStringCSVToMatrixObject
public static MatrixObject javaRDDStringCSVToMatrixObject(org.apache.spark.api.java.JavaRDD<String> javaRDD)
Convert aJavaRDD<String>in CSV format to aMatrixObject- Parameters:
javaRDD- the Java RDD of strings- Returns:
- the
JavaRDD<String>converted to aMatrixObject
-
javaRDDStringCSVToMatrixObject
public static MatrixObject javaRDDStringCSVToMatrixObject(org.apache.spark.api.java.JavaRDD<String> javaRDD, MatrixMetadata matrixMetadata)
Convert aJavaRDD<String>in CSV format to aMatrixObject- Parameters:
javaRDD- the Java RDD of stringsmatrixMetadata- matrix metadata- Returns:
- the
JavaRDD<String>converted to aMatrixObject
-
javaRDDStringCSVToFrameObject
public static FrameObject javaRDDStringCSVToFrameObject(org.apache.spark.api.java.JavaRDD<String> javaRDD)
Convert aJavaRDD<String>in CSV format to aFrameObject- Parameters:
javaRDD- the Java RDD of strings- Returns:
- the
JavaRDD<String>converted to aFrameObject
-
javaRDDStringCSVToFrameObject
public static FrameObject javaRDDStringCSVToFrameObject(org.apache.spark.api.java.JavaRDD<String> javaRDD, FrameMetadata frameMetadata)
Convert aJavaRDD<String>in CSV format to aFrameObject- Parameters:
javaRDD- the Java RDD of stringsframeMetadata- frame metadata- Returns:
- the
JavaRDD<String>converted to aFrameObject
-
javaRDDStringIJVToMatrixObject
public static MatrixObject javaRDDStringIJVToMatrixObject(org.apache.spark.api.java.JavaRDD<String> javaRDD, MatrixMetadata matrixMetadata)
Convert aJavaRDD<String>in IJV format to aMatrixObject. Note that metadata is required for IJV format.- Parameters:
javaRDD- the Java RDD of stringsmatrixMetadata- matrix metadata- Returns:
- the
JavaRDD<String>converted to aMatrixObject
-
javaRDDStringIJVToFrameObject
public static FrameObject javaRDDStringIJVToFrameObject(org.apache.spark.api.java.JavaRDD<String> javaRDD, FrameMetadata frameMetadata)
Convert aJavaRDD<String>in IJV format to aFrameObject. Note that metadata is required for IJV format.- Parameters:
javaRDD- the Java RDD of stringsframeMetadata- frame metadata- Returns:
- the
JavaRDD<String>converted to aFrameObject
-
rddStringCSVToMatrixObject
public static MatrixObject rddStringCSVToMatrixObject(org.apache.spark.rdd.RDD<String> rdd)
Convert aRDD<String>in CSV format to aMatrixObject- Parameters:
rdd- the RDD of strings- Returns:
- the
RDD<String>converted to aMatrixObject
-
rddStringCSVToMatrixObject
public static MatrixObject rddStringCSVToMatrixObject(org.apache.spark.rdd.RDD<String> rdd, MatrixMetadata matrixMetadata)
Convert aRDD<String>in CSV format to aMatrixObject- Parameters:
rdd- the RDD of stringsmatrixMetadata- matrix metadata- Returns:
- the
RDD<String>converted to aMatrixObject
-
rddStringCSVToFrameObject
public static FrameObject rddStringCSVToFrameObject(org.apache.spark.rdd.RDD<String> rdd)
Convert aRDD<String>in CSV format to aFrameObject- Parameters:
rdd- the RDD of strings- Returns:
- the
RDD<String>converted to aFrameObject
-
rddStringCSVToFrameObject
public static FrameObject rddStringCSVToFrameObject(org.apache.spark.rdd.RDD<String> rdd, FrameMetadata frameMetadata)
Convert aRDD<String>in CSV format to aFrameObject- Parameters:
rdd- the RDD of stringsframeMetadata- frame metadata- Returns:
- the
RDD<String>converted to aFrameObject
-
rddStringIJVToMatrixObject
public static MatrixObject rddStringIJVToMatrixObject(org.apache.spark.rdd.RDD<String> rdd, MatrixMetadata matrixMetadata)
Convert aRDD<String>in IJV format to aMatrixObject. Note that metadata is required for IJV format.- Parameters:
rdd- the RDD of stringsmatrixMetadata- matrix metadata- Returns:
- the
RDD<String>converted to aMatrixObject
-
rddStringIJVToFrameObject
public static FrameObject rddStringIJVToFrameObject(org.apache.spark.rdd.RDD<String> rdd, FrameMetadata frameMetadata)
Convert aRDD<String>in IJV format to aFrameObject. Note that metadata is required for IJV format.- Parameters:
rdd- the RDD of stringsframeMetadata- frame metadata- Returns:
- the
RDD<String>converted to aFrameObject
-
matrixObjectToJavaRDDStringCSV
public static org.apache.spark.api.java.JavaRDD<String> matrixObjectToJavaRDDStringCSV(MatrixObject matrixObject)
Convert aMatrixObjectto aJavaRDD<String>in CSV format.- Parameters:
matrixObject- theMatrixObject- Returns:
- the
MatrixObjectconverted to aJavaRDD<String>
-
frameObjectToJavaRDDStringCSV
public static org.apache.spark.api.java.JavaRDD<String> frameObjectToJavaRDDStringCSV(FrameObject frameObject, String delimiter)
Convert aFrameObjectto aJavaRDD<String>in CSV format.- Parameters:
frameObject- theFrameObjectdelimiter- the delimiter- Returns:
- the
FrameObjectconverted to aJavaRDD<String>
-
matrixObjectToJavaRDDStringIJV
public static org.apache.spark.api.java.JavaRDD<String> matrixObjectToJavaRDDStringIJV(MatrixObject matrixObject)
Convert aMatrixObjectto aJavaRDD<String>in IJV format.- Parameters:
matrixObject- theMatrixObject- Returns:
- the
MatrixObjectconverted to aJavaRDD<String>
-
frameObjectToJavaRDDStringIJV
public static org.apache.spark.api.java.JavaRDD<String> frameObjectToJavaRDDStringIJV(FrameObject frameObject)
Convert aFrameObjectto aJavaRDD<String>in IJV format.- Parameters:
frameObject- theFrameObject- Returns:
- the
FrameObjectconverted to aJavaRDD<String>
-
matrixObjectToRDDStringIJV
public static org.apache.spark.rdd.RDD<String> matrixObjectToRDDStringIJV(MatrixObject matrixObject)
Convert aMatrixObjectto aRDD<String>in IJV format.- Parameters:
matrixObject- theMatrixObject- Returns:
- the
MatrixObjectconverted to aRDD<String>
-
frameObjectToRDDStringIJV
public static org.apache.spark.rdd.RDD<String> frameObjectToRDDStringIJV(FrameObject frameObject)
Convert aFrameObjectto aRDD<String>in IJV format.- Parameters:
frameObject- theFrameObject- Returns:
- the
FrameObjectconverted to aRDD<String>
-
matrixObjectToRDDStringCSV
public static org.apache.spark.rdd.RDD<String> matrixObjectToRDDStringCSV(MatrixObject matrixObject)
Convert aMatrixObjectto aRDD<String>in CSV format.- Parameters:
matrixObject- theMatrixObject- Returns:
- the
MatrixObjectconverted to aRDD<String>
-
frameObjectToRDDStringCSV
public static org.apache.spark.rdd.RDD<String> frameObjectToRDDStringCSV(FrameObject frameObject, String delimiter)
Convert aFrameObjectto aRDD<String>in CSV format.- Parameters:
frameObject- theFrameObjectdelimiter- the delimiter- Returns:
- the
FrameObjectconverted to aRDD<String>
-
matrixObjectToListStringCSV
public static List<String> matrixObjectToListStringCSV(MatrixObject matrixObject)
Convert aMatrixObjectto aList<String>in CSV format.- Parameters:
matrixObject- theMatrixObject- Returns:
- the
MatrixObjectconverted to aList<String>
-
frameObjectToListStringCSV
public static List<String> frameObjectToListStringCSV(FrameObject frameObject, String delimiter)
Convert aFrameObjectto aList<String>in CSV format.- Parameters:
frameObject- theFrameObjectdelimiter- the delimiter- Returns:
- the
FrameObjectconverted to aList<String>
-
matrixObjectToListStringIJV
public static List<String> matrixObjectToListStringIJV(MatrixObject matrixObject)
Convert aMatrixObjectto aList<String>in IJV format.- Parameters:
matrixObject- theMatrixObject- Returns:
- the
MatrixObjectconverted to aList<String>
-
frameObjectToListStringIJV
public static List<String> frameObjectToListStringIJV(FrameObject frameObject)
Convert aFrameObjectto aList<String>in IJV format.- Parameters:
frameObject- theFrameObject- Returns:
- the
FrameObjectconverted to aList<String>
-
matrixObjectTo2DDoubleArray
public static double[][] matrixObjectTo2DDoubleArray(MatrixObject matrixObject)
Convert aMatrixObjectto a two-dimensional double array.- Parameters:
matrixObject- theMatrixObject- Returns:
- the
MatrixObjectconverted to adouble[][]
-
matrixObjectToDataFrame
public static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> matrixObjectToDataFrame(MatrixObject matrixObject, SparkExecutionContext sparkExecutionContext, boolean isVectorDF)
Convert aMatrixObjectto aDataFrame.- Parameters:
matrixObject- theMatrixObjectsparkExecutionContext- the Spark execution contextisVectorDF- is the DataFrame a vector DataFrame?- Returns:
- the
MatrixObjectconverted to aDataFrame
-
frameObjectToDataFrame
public static org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> frameObjectToDataFrame(FrameObject frameObject, SparkExecutionContext sparkExecutionContext)
Convert aFrameObjectto aDataFrame.- Parameters:
frameObject- theFrameObjectsparkExecutionContext- the Spark execution context- Returns:
- the
FrameObjectconverted to aDataFrame
-
matrixObjectToBinaryBlocks
public static org.apache.spark.api.java.JavaPairRDD<MatrixIndexes,MatrixBlock> matrixObjectToBinaryBlocks(MatrixObject matrixObject, SparkExecutionContext sparkExecutionContext)
Convert aMatrixObjectto aJavaPairRDD<MatrixIndexes, MatrixBlock>.- Parameters:
matrixObject- theMatrixObjectsparkExecutionContext- the Spark execution context- Returns:
- the
MatrixObjectconverted to aJavaPairRDD<MatrixIndexes, MatrixBlock>
-
frameObjectToBinaryBlocks
public static org.apache.spark.api.java.JavaPairRDD<Long,FrameBlock> frameObjectToBinaryBlocks(FrameObject frameObject, SparkExecutionContext sparkExecutionContext)
Convert aFrameObjectto aJavaPairRDD<Long, FrameBlock>.- Parameters:
frameObject- theFrameObjectsparkExecutionContext- the Spark execution context- Returns:
- the
FrameObjectconverted to aJavaPairRDD<Long, FrameBlock>
-
frameObjectTo2DStringArray
public static String[][] frameObjectTo2DStringArray(FrameObject frameObject)
Convert aFrameObjectto a two-dimensional string array.- Parameters:
frameObject- theFrameObject- Returns:
- the
FrameObjectconverted to aString[][]
-
jsc
public static org.apache.spark.api.java.JavaSparkContext jsc()
Obtain JavaSparkContext from MLContextProxy.- Returns:
- the Java Spark Context
-
sc
public static org.apache.spark.SparkContext sc()
Obtain SparkContext from MLContextProxy.- Returns:
- the Spark Context
-
spark
public static org.apache.spark.sql.SparkSession spark()
Obtain SparkSession from MLContextProxy.- Returns:
- the Spark Session
-
-