Skip to content

Escape UTF-8 dependent tests #7333

Description

@jangorecki

Currently out vanilla environment job, that does not have utf8 encoding:

* checking code files for non-ASCII characters ... OK
* checking R files for syntax errors ... WARNING
Warning in Sys.setlocale("LC_CTYPE", "en_US.UTF-8") :
  OS reports request to set locale to "en_US.UTF-8" cannot be honored

Is failing on the following tests:


  Test 2194.7 produced 0 errors but expected 1
  Expected: Internal error.*types or lengths incorrect
  Observed: 
  Tests 2253* skipped because they need a UTF-8 locale.
  Tests 2266* skipped. n-tilde cannot be represented in the native encoding on this system. Please report to the data.table issue tracker.
  Test 2298.1 ran without errors but failed check that x equals y:
  > x = rbindlist(list(x, y), use.names = TRUE) 
     <U+00E4> <U+00F6> <U+00FC>  [Key= Types=dou,dou,dou Classes=num,num,num]
  1:        1        2        3                                              
  2:        6        4        5                                              
  > y = data.table(`<U+00E4>` = c(1, 6), `<U+00F6>` = c(2, 4), `<U+00FC>` = c(3,      5)) 
     <U+00E4> <U+00F6> <U+00FC>  [Key= Types=dou,dou,dou Classes=num,num,num]
  1:        1        2        3                                              
  2:        6        4        5                                              
  Different column names
  Test 2298.2 ran without errors but failed check that x equals y:
  > x = rbindlist(list(y, x), use.names = TRUE) 
     <f6> <fc> <e4>  [Key= Types=dou,dou,dou Classes=num,num,num]
  1:    4    5    6                                              
  2:    2    3    1                                              
  > y = data.table(`<U+00F6>` = c(4, 2), `<U+00FC>` = c(5, 3), `<U+00E4>` = c(6,      1)) 
     <U+00F6> <U+00FC> <U+00E4>  [Key= Types=dou,dou,dou Classes=num,num,num]
  1:        4        5        6                                              
  2:        2        3        1                                              
  Different column names
  Test 2298.3 ran without errors but failed check that x equals y:
  > x = rbindlist(list(x, y), use.names = TRUE, fill = TRUE) 
     <U+00E4> <U+00F6> <U+00FC>  [Key= Types=dou,dou,dou Classes=num,num,num]
  1:        1        2        3                                              
  2:       NA        4        5                                              
  > y = data.table(`<U+00E4>` = c(1, NA), `<U+00F6>` = c(2, 4), `<U+00FC>` = c(3,      5)) 
     <U+00E4> <U+00F6> <U+00FC>  [Key= Types=dou,dou,dou Classes=num,num,num]
  1:        1        2        3                                              
  2:       NA        4        5                                              
  Different column names
  Test 2298.4 ran without errors but failed check that x equals y:
  > x = rbindlist(list(y, x), use.names = TRUE, fill = TRUE) 
     <f6> <fc> <U+00E4>  [Key= Types=dou,dou,dou Classes=num,num,num]
  1:    4    5       NA                                              
  2:    2    3        1                                              
  > y = data.table(`<U+00F6>` = c(4, 2), `<U+00FC>` = c(5, 3), `<U+00E4>` = c(NA,      1)) 
     <U+00F6> <U+00FC> <U+00E4>  [Key= Types=dou,dou,dou Classes=num,num,num]
  1:        4        5       NA                                              
  2:        2        3        1                                              
  Different column names
  V1
  1
  V1
  0,1
  0,2
  a,b
  a
  
  Sun Sep 21 04:23:35 2025  endian==little, sizeof(long double)==16, capabilities('long.double')==TRUE, longdouble.digits==64, sizeof(pointer)==8, TZ=='UTC', Sys.timezone()=='UTC', Sys.getlocale()=='C', l10n_info()=='MBCS=FALSE; UTF-8=FALSE; Latin-1=FALSE; codeset=ANSI_X3.4-1968', getDTthreads()=='This installation of data.table has not been compiled with OpenMP support.; omp_get_num_procs()==1; R_DATATABLE_NUM_PROCS_PERCENT==unset (default 50); R_DATATABLE_NUM_THREADS==unset; R_DATATABLE_THROTTLE==unset (default 1024); omp_get_thread_limit()==1; omp_get_max_threads()==1; OMP_THREAD_LIMIT==unset; OMP_NUM_THREADS==unset; RestoreAfterFork==true; data.table is using 1 threads with throttle==1024. See ?setDTthreads.', .libPaths()=='/builds/Rdatatable/data.table/data.table.Rcheck','/usr/local/lib/R/site-library','/usr/lib/R/site-library','/usr/lib/R/library', zlib header files were not found when data.table was compiled
  Error in test.data.table() : 
    5 errors out of 10961. Search tests/tests.Rraw for test numbers 2194.7, 2298.1, 2298.2, 2298.3, 2298.4. Duration: 27.5s elapsed (27.2s cpu).

It could use same escape as:

Tests 2253* skipped because they need a UTF-8 locale.
  Tests 2266* skipped. n-tilde cannot be represented in the native encoding on this system. Please report to the data.table issue tracker.

So we don't raise false positive failure.

Activity

  1. jangorecki commented on Sep 23, 2025

    @jangorecki
    MemberAuthor

    More of the code, probably around tests that were escaped here, has to be escaped as well: https://rdatatable.gitlab.io/data.table/web/checks/data.table/test-lin-rel-vanilla/00check.log

    Running the tests in 'tests/main.R' failed.
    Complete output:
      > require(data.table)
      Loading required package: data.table
      > 
      > test.data.table()  # runs the main test suite of 5,000+ tests in /inst/tests/tests.Rraw
      getDTthreads(verbose=TRUE):
      This installation of data.table has not been compiled with OpenMP support.
        omp_get_num_procs()            1
        R_DATATABLE_NUM_PROCS_PERCENT  unset (default 50)
        R_DATATABLE_NUM_THREADS        unset
        R_DATATABLE_THROTTLE           unset (default 1024)
        omp_get_thread_limit()         1
        omp_get_max_threads()          1
        OMP_THREAD_LIMIT               unset
        OMP_NUM_THREADS                unset
        RestoreAfterFork               true
        data.table is using 1 threads with throttle==1024. See ?setDTthreads.
      test.data.table() running: /builds/Rdatatable/data.table/data.table.Rcheck/data.table/tests/tests.Rraw
      
      **** Suggested package bit64 is not installed or has dependencies missing. Tests using it will be skipped.
      
      
      **** Suggested package R.utils is not installed or has dependencies missing. Tests using it will be skipped.
      
      Test 2194.7 skipped because it needs a UTF-8 locale.
      Tests 2253* skipped because they need a UTF-8 locale.
      Tests 2266* skipped. n-tilde cannot be represented in the native encoding on this system. Please report to the data.table issue tracker.
      Tests 2298.* skipped because they need a UTF-8 locale.
      V1
      1
      V1
      0,1
      0,2
      a,b
      a
      
      Wed Sep 24 04:24:14 2025  endian==little, sizeof(long double)==16, capabilities('long.double')==TRUE, longdouble.digits==64, sizeof(pointer)==8, TZ=='UTC', Sys.timezone()=='UTC', Sys.getlocale()=='C', l10n_info()=='MBCS=FALSE; UTF-8=FALSE; Latin-1=FALSE; codeset=ANSI_X3.4-1968', getDTthreads()=='This installation of data.table has not been compiled with OpenMP support.; omp_get_num_procs()==1; R_DATATABLE_NUM_PROCS_PERCENT==unset (default 50); R_DATATABLE_NUM_THREADS==unset; R_DATATABLE_THROTTLE==unset (default 1024); omp_get_thread_limit()==1; omp_get_max_threads()==1; OMP_THREAD_LIMIT==unset; OMP_NUM_THREADS==unset; RestoreAfterFork==true; data.table is using 1 threads with throttle==1024. See ?setDTthreads.', .libPaths()=='/builds/Rdatatable/data.table/data.table.Rcheck','/usr/local/lib/R/site-library','/usr/lib/R/site-library','/usr/lib/R/library', zlib header files were not found when data.table was compiled
      Caught 13 warnings outside the test() calls:
          after test
               <num>
       1:       -1.0
       2:       -1.0
       3:       -1.0
       4:       -1.0
       5:       -1.0
       6:       -1.0
       7:       -1.0
       8:       -1.0
       9:       -1.0
      10:       -1.0
      11:       -1.0
      12:       -1.0
      13:     2194.6
                                                                                                                                 warning
                                                                                                                                  <char>
       1:                                                                              unable to translate '<U+00E4>' to native encoding
       2:                                                                              unable to translate '<U+00F6>' to native encoding
       3:                                                                              unable to translate '<U+00FC>' to native encoding
       4:                                                                              unable to translate '<U+00F6>' to native encoding
       5:                                                                              unable to translate '<U+00FC>' to native encoding
       6:                                                                              unable to translate '<U+00E4>' to native encoding
       7:                                                                              unable to translate '<U+00E4>' to native encoding
       8:                                                                              unable to translate '<U+00F6>' to native encoding
       9:                                                                              unable to translate '<U+00FC>' to native encoding
      10:                                                                              unable to translate '<U+00F6>' to native encoding
      11:                                                                              unable to translate '<U+00FC>' to native encoding
      12:                                                                              unable to translate '<U+00E4>' to native encoding
      13: invalid input found on input connection '/builds/Rdatatable/data.table/data.table.Rcheck/data.table/tests/issue_563_fread.txt'
                                                                                                                                                                                                                                           calls
                                                                                                                                                                                                                                          <char>
       1:                     test.data.table -> try -> tryCatch -> tryCatchList -> tryCatchOne -> doTryCatch -> withCallingHandlers -> sys.source -> parse -> .signalSimpleWarning -> withRestarts -> withOneRestart -> doWithOneRestart -> ...
       2:                     test.data.table -> try -> tryCatch -> tryCatchList -> tryCatchOne -> doTryCatch -> withCallingHandlers -> sys.source -> parse -> .signalSimpleWarning -> withRestarts -> withOneRestart -> doWithOneRestart -> ...
       3:                     test.data.table -> try -> tryCatch -> tryCatchList -> tryCatchOne -> doTryCatch -> withCallingHandlers -> sys.source -> parse -> .signalSimpleWarning -> withRestarts -> withOneRestart -> doWithOneRestart -> ...
       4:                     test.data.table -> try -> tryCatch -> tryCatchList -> tryCatchOne -> doTryCatch -> withCallingHandlers -> sys.source -> parse -> .signalSimpleWarning -> withRestarts -> withOneRestart -> doWithOneRestart -> ...
       5:                     test.data.table -> try -> tryCatch -> tryCatchList -> tryCatchOne -> doTryCatch -> withCallingHandlers -> sys.source -> parse -> .signalSimpleWarning -> withRestarts -> withOneRestart -> doWithOneRestart -> ...
       6:                     test.data.table -> try -> tryCatch -> tryCatchList -> tryCatchOne -> doTryCatch -> withCallingHandlers -> sys.source -> parse -> .signalSimpleWarning -> withRestarts -> withOneRestart -> doWithOneRestart -> ...
       7:                     test.data.table -> try -> tryCatch -> tryCatchList -> tryCatchOne -> doTryCatch -> withCallingHandlers -> sys.source -> parse -> .signalSimpleWarning -> withRestarts -> withOneRestart -> doWithOneRestart -> ...
       8:                     test.data.table -> try -> tryCatch -> tryCatchList -> tryCatchOne -> doTryCatch -> withCallingHandlers -> sys.source -> parse -> .signalSimpleWarning -> withRestarts -> withOneRestart -> doWithOneRestart -> ...
       9:                     test.data.table -> try -> tryCatch -> tryCatchList -> tryCatchOne -> doTryCatch -> withCallingHandlers -> sys.source -> parse -> .signalSimpleWarning -> withRestarts -> withOneRestart -> doWithOneRestart -> ...
      10:                     test.data.table -> try -> tryCatch -> tryCatchList -> tryCatchOne -> doTryCatch -> withCallingHandlers -> sys.source -> parse -> .signalSimpleWarning -> withRestarts -> withOneRestart -> doWithOneRestart -> ...
      11:                     test.data.table -> try -> tryCatch -> tryCatchList -> tryCatchOne -> doTryCatch -> withCallingHandlers -> sys.source -> parse -> .signalSimpleWarning -> withRestarts -> withOneRestart -> doWithOneRestart -> ...
      12:                     test.data.table -> try -> tryCatch -> tryCatchList -> tryCatchOne -> doTryCatch -> withCallingHandlers -> sys.source -> parse -> .signalSimpleWarning -> withRestarts -> withOneRestart -> doWithOneRestart -> ...
      13: test.data.table -> try -> tryCatch -> tryCatchList -> tryCatchOne -> doTryCatch -> withCallingHandlers -> sys.source -> eval -> eval -> readLines -> .signalSimpleWarning -> withRestarts -> withOneRestart -> doWithOneRestart -> ...
      Error in test.data.table() : 
        Tests succeeded, but non-test code caused warnings. Search tests/tests.Rraw for tests shown above.
      Calls: test.data.table -> stopf -> raise_condition -> signal
      Execution halted
    * DONE
    Status: 1 ERROR, 1 WARNING, 1 NOTE
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    encodingissues related to Encodingtests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions