Sqoop1 Import Job Failed With Error “java.io.IOException: No columns to generate for ClassWriter”
Cloudera

Sqoop1 Import Job Failed With Error “java.io.IOException: No columns to generate for ClassWriter”

Cloudera
Recently when I was testing Sqoop1 command in my CDH cluster, I kept getting “java.io.IOException: No columns to generate for ClassWriter” error. The full command was like below: 
sqoop import --connect jdbc:mysql:///test 
    --table test 
    --username  
    --password  
    --target-dir sqoop_test 
    -m 1
And full stacktrace: 
16/08/20 03:03:13 ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@7cd1be26 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@7cd1be26 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
	at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2735)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1899)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2569)
	at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1524)
	at com.mysql.jdbc.ConnectionImpl.getMaxBytesPerChar(ConnectionImpl.java:3003)
	at com.mysql.jdbc.Field.getMaxBytesPerCharacter(Field.java:602)
	at com.mysql.jdbc.ResultSetMetaData.getPrecision(ResultSetMetaData.java:445)
	at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:305)
	at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260)
	at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:246)
	at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:327)
	at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1846)
	at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1646)
	at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
	at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488)
	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
	at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
16/08/20 03:03:13 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
	at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1652)
	at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
	at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488)
	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
	at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Although I do not have the RCA yet for this issue, I do have a workaround, which is adding “––driver com.mysql.jdbc.Driver” to the Sqoop parameters. So the full command becomes: 
sqoop import --connect jdbc:mysql:///test 
    --table test 
    --username  
    --password  
    --target-dir sqoop_test 
    -m 1 
    --driver com.mysql.jdbc.Driver
Hopefully this can help with anyone who might have the same issue.

Other posts that you might also be interested:

  1. Sqoop Job Failed with Error: java.lang.NoClassDefFoundError: org/json/JSONObject
  2. Sqoop job failed with “Malformed option in options file” error
  3. Oozie Server failed to Start with error java.lang.NoSuchFieldError: EXTERNAL_PROPERTY
  4. Sqoop Hive Import Failed After Upgrading to CDH5.4.x or CDH5.5.x
  5. HiveMetaStore Failed to Start in Cloudera Manager: cmf.service.config.ConfigGenException: Unable to generate config file creds.localjceks
  6. Unable to Import Data as Parquet into Encrypted HDFS Zone | Sqoop Parquet Import

Powered by YARPP.

12 Comments

  3. arun singh
    Reply

    Hi I am new in bigdata and i am using –>sqoop import –connect jdbc:mysql://xxx.xxx.xx.xx:3306/FBI_DB –table FBI_CRIME –hbase-table H_FBI_CRIME –column-family cd –hbase-row-key ID –m 1 –username root -P. But I am not able to import data. Once I had used (––driver com.mysql.jdbc.Driver) but still didn’t get success . Always showing error —->>>ERROR tool.ImportTool: Import failed: java.io.IOException: No columns to generate for ClassWriter.

    Please help, What is wrong.

    1. Eric Lin
      Reply

      Hi Arun,

      Thanks for visiting my blog and posting question. In order to find out why it is still not working for you, can you please share the full –verbose output of your Sqoop command output and email to my address eric at lin dot fit. I will need to check the full error details to be able to determine what else to look for.

      Cheers

      1. christian
        Reply

        Hi Eric

        my error still same “ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter”

        and this is my syntax for import with sqoop “sqoop import –connect jdbc:mysql://localhost/movielens –driver com.mysql.jdbc.Driver –table movies -m 1
        ”
        can you help me how to solve this error , thanks…

        1. Eric Lin
          Reply

          Hi Christian,

          Can you try to below changes:

          1. from “localhost” to IP or hostname?
          2. you do not have username and password for the mysql connection, please add them

          Please also share the errors before “No columns to generate for ClassWriter”, as there might be other useful info there as well.

          Cheers
          Eric

  5. Samira Karki
    Reply

    hi Eric

    I am a newbie in sqoop, came across your blog, I am hoping you could help me resolve this issue.

    I have a table in MySQL with the primary key as varchar which is carId, importing it to Hadoop with mapper = 5.

    –command–
    sqoop import –connect jdbc:mysql://sqoopdb.slbdh.cloudlabs.com:3306/ –username –password –table stg_Cars_Classified_Ads –target-dir /user//cars/mysql_hadoop -m 5

    It gives error “Caused by: Generating splits for a textual index column allowed only in case of “-Dorg.apache.sqoop.splitter.allow_text_splitter=true” property passed as a parameter”.

    I tried adding -Dorg.apache.sqoop.splitter.allow_text_splitter=true in the command , it gives another error of “Import failed: java.io.IOException: No columns to generate for ClassWriter”.

    I believe it is because the carId is varchar and not int, how do you go about fixing it.

    1. Eric Lin
      Reply

      Hi Samira,

      Thanks for visiting my blog and posting questions.

      Can you add “–verbose” to the Sqoop command and see if you can get more details about the error? I realise “No columns to generate for ClassWriter” is quite generic and can happen by many different causes.

      Cheers
      Eric

      1. Samira Karki
        Reply

        Hello Eric,

        It took me while to get back. The error says, connection refused.
        I have tried researching a lot, but could not understand.
        I have the error message below.
        Thank you,
        samira

        SLF4J: Class path contains multiple SLF4J bindings.
        SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
        SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
        SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
        SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
        21/02/17 04:42:47 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7-cdh6.3.2
        21/02/17 04:42:47 DEBUG tool.BaseSqoopTool: Enabled debug logging.
        21/02/17 04:42:47 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
        21/02/17 04:42:47 DEBUG tool.BaseSqoopTool: Parquet job configurator implementation set: HADOOP
        21/02/17 04:42:47 DEBUG sqoop.ConnFactory: Loaded manager factory: org.apache.sqoop.manager.oracle.OraOopManagerFactory
        21/02/17 04:42:47 DEBUG sqoop.ConnFactory: Loaded manager factory: org.apache.sqoop.manager.DefaultManagerFactory
        21/02/17 04:42:47 DEBUG sqoop.ConnFactory: Trying ManagerFactory: org.apache.sqoop.manager.oracle.OraOopManagerFactory
        21/02/17 04:42:47 DEBUG oracle.OraOopUtilities: Enabled OraOop debug logging.
        21/02/17 04:42:47 DEBUG oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop can be called by Sqoop!
        21/02/17 04:42:47 DEBUG sqoop.ConnFactory: Trying ManagerFactory: org.apache.sqoop.manager.DefaultManagerFactory
        21/02/17 04:42:47 DEBUG manager.DefaultManagerFactory: Trying with scheme: jdbc:mysql:
        21/02/17 04:42:47 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
        21/02/17 04:42:47 DEBUG sqoop.ConnFactory: Instantiated ConnManager org.apache.sqoop.manager.MySQLManager@548a102f
        21/02/17 04:42:47 INFO tool.CodeGenTool: Beginning code generation
        21/02/17 04:42:47 DEBUG manager.SqlManager: Execute getColumnInfoRawQuery : SELECT t.* FROM `stg_Cars_Classified_Ads` AS t LIMIT 1
        Loading class `com.mysql.jdbc.Driver’. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver’. The driver is automatically registered via the SPI a
        nd manual loading of the driver class is generally unnecessary.
        21/02/17 04:42:47 DEBUG manager.SqlManager: No connection paramenters specified. Using regular API for making connection.
        21/02/17 04:42:47 ERROR manager.SqlManager: Error executing statement: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
        The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
        The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)

        1. Eric Lin
          Reply

          I am really sorry Samira, I over looked your reply.

          Based on the error “Communications link failure”, it looks like driver was not able to reach to the MySQL server, have you tried and tested that all hosts in the cluster can reach the MySQL server?

          Cheers
          Eric

Leave a Reply

Your email address will not be published.

✕

My new Snowflake Blog is now live. I will not be updating this blog anymore but will continue with new contents in the Snowflake world!