Cluster bricks when all journalnodes are down #338

Activity

1. nightkr

   added

   type/bug

   on Mar 30, 2023
2. lfrancke

   added

   priority/high

   on Mar 30, 2023
3. lfrancke

   moved this to Next in Stackable End-to-End Coordinationon Mar 30, 2023
4. lfrancke

   added this to Stackable End-to-End Coordinationon Mar 30, 2023
5. sbernauer commented on Jun 1, 2023

   

   sbernauer

   on Jun 1, 2023

   Member

   More actions

   Good news:
   IIRC this happened when demo-ing Kerberos support, I did also run into this problem.
   The current WIP state of hdfs Kerberos has solved the problem and added kuttl tests to test this scenario. Linking the PR als fix
6. sbernauer

   added this to Stackable Engineering and removed this from Stackable End-to-End Coordinationon Jun 1, 2023
7. sbernauer

   moved this to Development: In Progress in Stackable Engineeringon Jun 1, 2023
8. sbernauer commented on Jun 5, 2023

   

   sbernauer

   on Jun 5, 2023

   Member

   More actions

   The problem was the journalnodes rejecting the namenodes because of reverse DNS roulette

   org.apache.hadoop.hdfs.server.common.HttpGetFailedException: Fetch of https://hdfs-journalnode-default-0.hdfs-journalnode-default.kuttl-test-fine-rat.svc.cluster.local:8481/getJournal?jid=hdfs&segmentTxId=1&storageInfo=-65%3A595659877%3A1685437352616%3ACID-90c52400-5b07-49bf-bdbe-3469bbdc5ebb&inProgressOk=true failed with status code 403
   Response message:
   Only Namenode and another JournalNode may access this servlet
   

   resulting in

   2023-05-30 09:23:01,651 ERROR namenode.RedundantEditLogInputStream (RedundantEditLogInputStream.java:nextOp(233)) - Got error reading edit log input stream https://hdfs-journalnode-default-0.hdfs-journalnode-default.kuttl-test-fine-rat.svc.cluster.local:8481/getJournal?jid=hdfs&segmentTxId=1&storageInfo=-65%3A595659877%3A1685437352616%3ACID-90c52400-5b07-49bf-bdbe-3469bbdc5ebb&inProgressOk=true; failing over to edit log https://hdfs-journalnode-default-1.hdfs-journalnode-default.kuttl-test-fine-rat.svc.cluster.local:8481/getJournal?jid=hdfs&segmentTxId=1&storageInfo=-65%3A595659877%3A1685437352616%3ACID-90c52400-5b07-49bf-bdbe-3469bbdc5ebb&inProgressOk=true
   

   The got premature end-of-file at txid 0; expected file to go up to 10 was because of the failover

   I solved the problem by not using _HOST in the principals, so that journalnode did not reject the namenode
9. sbernauer commented on Jun 5, 2023

   

   sbernauer

   on Jun 5, 2023

   Member

   More actions

   Closing this, as we fixed the problem in the Kerberos feature-branch. Please feel free to re-open when the problem reappears!
10. sbernauer

    closed this as completedon Jun 5, 2023
11. sbernauer

    moved this from Development: In Progress to Development: Done in Stackable Engineeringon Jun 6, 2023
12. lfrancke

    added

    release/23.7.0

    on Jun 9, 2023
13. lfrancke

    moved this from Development: Done to Done in Stackable Engineeringon Jun 9, 2023
14. bors

    added a commit that references this issue on Jun 14, 2023

    Allow enabling secure mode with Kerberos (#334)

    ...

    ef4d433