I am following the official documentation for procedure for Restoring a snapshot into a new cluster. As per this,
4. Clear the system table data from each new node:
$ sudo rm -rf /var/lib/cassandra/data/system/*
This allows the new nodes to use the initial tokens defined in the cassandra.yaml when they restart.
5. Start each node using the specified list of token ranges in new cluster's cassandra.yaml:
initial_token: -9211270970129494930, -9138351317258731895, -8980763462514965928, ...
I am following these 2 steps exactly. In fact, it is verified by cassandra debug logs.
INFO [main] 2017-11-16 09:36:59,353 Config.java:456 - Node configuration:[allocate_tokens_for_keyspace=null; authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=true; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=null; broadcast_rpc_address=null; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=cassandra-staging-restored; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/mnt/cassandra/data/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=null; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@43301423; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=SimpleSnitch; file_cache_size_in_mb=null; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=/mnt/cassandra/data/hints; hints_flush_period_in_ms=10000; incremental_backups=true; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=-9095105683159122588, -9064001015036599926, -9013000709875877114, -8829167773634478741, -8730304531409075421, -8615439886540673174, -8270623326447012978, -8176731878564157082, -8106822260307184341, -8089230042068892584, -8048906315856045311, -8028060929375944874, -7997403222481000091, -7980484429523277359, -7974470050771642191, -7851931228156318619, -7803664353182134419, -7788144370469500201, -7757626955856652301, -7712114049802319218, -7567720775920874700, -7551341429242695915, -7541865877881094180, -7514992388769119302, -7453108576041420033, -7431137934229111567, -7385697214426296194, -7326461701752475949, -7229952075030336177, -7218589485893294366, -7135605539700697459, -7062493953257887070, -6920687217081027707, -6916831669158286277, -6834053763268440089, -6822142837027528938, -6803094670398379101, -6602427539566207665, -6568182869870496307, -6511459920765505444, -6475249533685490199, -6428402884562770941, -6404676233544011759, -6360685224996369272, -6247121131610539884, -6246197555819424550, -6237368585021940017, -6153182247036415095, -6111450384148334970, -6042018771685803203, -5944793473876873149, -5863379160236916223, -5856475313797546183, -5832717016324856051, -5638468189438508210, -5539167370710819367, -5485280601523987503, -5474309089162218374, -5319174053461867928, -5300518577149097037, -5245536441075283612, -5112569688111722752, -4898240302203107111, -4881568115746221858, -4824104622873342498, -4701179017811291386, -4429755340063878775, -4413214557193145518, -4345418978114596882, -4079966796765885842, -4061762443047240866, -4030858175577795653, -4021547736638981847, -3836302740108504981, -3751768720526789019, -3743461742087956083, -3608810097734181459, -3607048178493709454, -3581158125752648978, -3562376827907111596, -3534668228830524620, -3431416556842048483, -3393885233868129949, -3339504094398301776, -3339170395338364082, -3307612236986384680, -3265920212434927166, -3204785075022793989, -3140386460827349755, -3099066887010465462, -3080689217189133335, -3007133744629835172, -2959846903344801620, -2888247723844507160, -2797992536166356195, -2712194459828626645, -2684177990525837322, -2658075701966969657, -2561917844478305786, -2492629295365129028, -2487719825222497546, -2399789247381255821, -2241309724624423586, -2108173119695457257, -2052360610564904726, -1939271054812802955, -1917043893384955146, -1808888302673267763, -1646863387940805021, -1634994825307804368, -1609712683106528653, -1300771381145801651, -1278385576576119381, -1274643898977022746, -950901630804463153, -940580146197028656, -731515730336327054, -642040964228514704, -608433393788453231, -478573951386945456, -436153095069462168, -416275106775833677, -378149959056786595, -346798880456427869, -305717523770204725, -284546652114461767, -166337978258482678, -101574744252082785, -30754271282451326, -6946162971722353, -4662052994008400, 68233019263256270, 126207333796953658, 132958619259436796, 166586855936325288, 180257471364193954, 222412061324634542, 299152219068289859, 307967555141655158, 499885456475913627, 510710569344453193, 512082871639239079, 551467999030970818, 560270740776978967, 563581994761305584, 687058605177782549, 793693682518320692, 917345500055208403, 935027642243141249, 1167582121991529631, 1177005535757550036, 1226282557885818866, 1279109759358271230, 1376609314865350358, 1388467321396881673, 1422156903349974253, 1475548349950097082, 1586261249441981681, 1594387271495404471, 1637548112971596569, 1759270488237130010, 1936016394983722984, 1991608383818464987, 2208451829114082571, 2242674193749084268, 2461080713362404466, 2539772799528407861, 2559844306686112571, 2620884963938532750, 2631650124112820870, 2664129588892760377, 2704157604972767845, 2736768902813611411, 2843328646543212128, 2856803091245137446, 2872960500638572538, 3138686478845054074, 3162031242206258542, 3185538412991024850, 3256233823832431553, 3284337403758100585, 3382411519818362465, 3490207566034869602, 3594317170903296648, 3669639735626576929, 3787200951518921417, 3805575545151951881, 3930156481299524524, 4109302677069830167, 4162318623147816383, 4174122441845031013, 4184778082562775012, 4333398546246868435, 4429402131470597299, 4454026097116561450, 4505802367873280245, 4538296977730990054, 4556607070107659214, 4570854427961759192, 4623296353148907992, 4679204413345951708, 4835621991534530902, 4873358208143831472, 4877348486234759943, 4909265243325179596, 4916889701925751547, 5050438954348872869, 5095687454372630280, 5225622553567840544, 5362122150810830638, 5453189930700137106, 5607985707233264385, 5627412929991624017, 5715642638971959215, 5792552742790972143, 5958282382337339668, 5966221173142140715, 6034708041737829912, 6035042265093059146, 6044918925766439026, 6117274929043661295, 6287231019632644302, 6317296677286381680, 6347246668613761526, 6386913449650684351, 6484548945192165912, 6809960070854586220, 7007749647532345769, 7064744556786092960, 7188015426890314535, 7381000932719845319, 7587261946507044657, 7642859702759477380, 7880162680349272441, 7908612477022491169, 7933323898613557131, 7937658449054724067, 7970141900392419694, 7979202070554210097, 7990763252197782986, 7995201022333707512, 7995792407943250493, 8140167553977835725, 8207869160118839667, 8384513993725731980, 8425627379584317458, 8450167920418959385, 8558753444678316624, 8635896206896058589, 8655999832148389735, 8749247593003798317, 8793336959659815954, 8811175299936105223, 8960172117500802825, 9082531635065196023, 9119004294127950942; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=dc; internode_recv_buff_size_in_bytes=null; internode_send_buff_size_in_bytes=null; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.120.8.116; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=1; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=256; otc_coalescing_strategy=TIMEHORIZON; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=10.120.8.116; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/mnt/cassandra/data/saved_caches; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=10.120.8.233,10.120.8.213,10.120.8.116,10.120.8.29,10.120.8.251}; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@2f112965; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000
But the initial_tokens are those which were auto-generated by cassandra when it was first started.
INFO [main] 2017-11-16 05:52:57,322 StorageService.java:1030 - Generated random tokens. tokens are [-1389648523687197109, -732698274772120137, -6376639320886405616, -873424736264959023, 8595953819512872578, 6773654626092013980, 9148695480551165081, 8209470968081970646, 9110581623169741801, 8012929209487966833, 7452692620314248468, 3103266257146948503, 5679525578987403003, -4522063905876778141, -6135750996189542432, 415784996037656153, 4396058137183394429, 692447961111726586, -6531662267543500154, 245834976461441926, -3064167806548065705, 8943572903844965496, -8638802229987964486, -8160083686661602566, -3050334815467563212, -8542758292854754936, 5709282354905481607, 88986928287276648, 1373249395890303055, -1686001724516419324, -396828971815045007, -8794909476111352967, -222280635739262458, -5414806845469275848, -2054698046151951295, -3554921400589962677, -7505704101012649583, 4179877325296378655, 7507175622338132461, -3496771293631065989, -3269044636044041612, -2009963074060585591, -2105986620005447634, 5330076034215385934, -4344640060181026104, 617483922231457714, 6465679396586550255, 5268797296416377890, 8377470147983205477, -8297832552754006887, -74853680046366003, -2872441811245832825, -2916068111566416056, 8408680662553618809, -2520357715711164569, -1560036757814314210, -5257702498481345613, -1017232662177787435, -4767292845107768798, -907567868062273876, 8870536721819084448, -6581232938993081706, 3822595822833559746, -7289338527755522826, -7639859898047414864, -4037700692557190365, 5049983552065810879, -5561302251870351386, 4621684495888003169, -3131443300514015633, -1160430662405286954, -8097490517397089231, -7348298090590090758, 5509208719656812112, -6481250052576231824, -5996741154396422167, -8294729397011283217, 2995257461252438837, -4352117679133792335, 1233012258201042528, 9035122281098363653, 6405344801897697404, 8708109275467472592, 8539949944069553390, 213833451306356755, -4166035949030694285, -8970221914082044097, 4606844533012524108, 947526220642208048, -2801978367402508153, -4532865176940942027, -1594392693891556555, -5806887334100689059, -2394088246665207424, 4234652354740925563, -8076001124699026366, 8744811137913444802, 62796481248292599, 6455733404981444252, -126533974378819127, 5843584371728823073, 5445168425290460940, 6707320880322895668, 4573751191752231090, -2127663589976413582, 1954401089317717447, -5300938135871221896, 5093721835155884661, -2524179837794403109, -6385219008186858160, 7325353601705788106, -840570978863751222, 5759885770816159939, 9072054183035599574, -8308763822241266759, -2586762965601191160, -432658258571974620, 3019044186650827069, 5726665958978111481, 6052672856134820217, 7838411235214044945, 2112902061202895279, 6260813920434498398, -8658490402119023263, 2928460241999085176, -8088631625855485121, -3893350039068301015, -1883547399497783790, -4931508961242998715, 1962026677878034467, -3536945581565756993, -3980665635466622441, 1616177331508810053, 4312023170834343978, 2878951844701050184, 7237763498599271588, -4143611656813097424, -3034833307460628079, -8631680602869413810, -5149492198682224552, -2010682875598030955, 2599132540851755500, -2955657629453800874, 7849160639412118736, 4066126615292533141, 8785492290797937036, -2425804637592817359, -81469586499474786, -8457085973377633413, 7938935767281129568, 5555025259044091280, -928104310141668027, 7576407531657016973, 1924068444759082739, -714923442136005577, -7424796994965823987, -8286550103460241539, 3795917762129273051, -8178762033443435991, 199935774895264752, 7223315731747941169, 2131952788031994270, -5924194093356500134, 8312801612720650647, -3455990260940501757, -6752550707935826850, -6601973406625917854, 1003897288641305994, -2311356538255289914, 6198153789687000292, 4371280172613277105, -3103750689457909819, -4749463815536259135, -371035516354562506, -3121297865532053724, 8982910635457246180, -1228260466727841004, -8870629066952254909, -526008540797435088, -5031238755845142021, -8363050244592607033, 1050601700640744024, -9104611786673170314, -4479945644147199241, -6823419782998328301, -6797181997798727850, 5481659148496729987, -8215699248520360452, -517420190806294229, 3787637764983639252, 2805777124248862676, 847375026463710908, -752806257333244860, -5802733257867073965, -6973496662755627239, 3473454836096478605, 5913384000201698403, -5389236760772073822, -8414280654740947122, 3781334199812295253, 7106186427969879811, -5793392653945612514, 4025496630557953873, 2029509328065975744, 7572757562784032698, -2409285778846785268, 8978472449360675819, 1502576153090782858, -524754924171314294, -5430506207796491830, -3681655679894962053, 5884494403462724943, 7057525460429085795, 6045128197566372326, 3068439635150742873, 5459483613324200529, -1099342907611602685, -8333623299114170297, 5575333565033227947, -8354986150955517165, 8957282480649403114, -4820459891421864876, 6809654053429862128, 4545870922630195580, 7197311542935486176, -2224608079234315120, -2141755264538606024, 2567830060494624486, -6270558622819592064, -4631885184728004763, 5954771488972332706, 7426073577700947604, 9036595665094295454, -549019315323478217, -7605262589627026961, 7538503311332390280, 5243496430380841675, -5125800136257737648, -189067622801948902, 6609318324048658940, -6407003720857192075, 6821243825376985989, -8663409582093444260, -8403726365964456251, -6966940567562465809, 2209597621040018613, -785009207590992232, 7148233497746128406, 8290593398169627427, -1934557347998048138, -1747811994016572301, -5360972975488153447, -4344532326911873111, -2181569417592666651, -5367930324551905863, -7161819751367151648]
What am I missing? Isn't it enough to specify initial_token and restart the cassandra?
I can't avoid random generation of initial_token by cassandra, because cassandra is automatically started after installation, i.e. even before I can edit cassandra.yaml for initial_token specification.
I am using cassandra 3.7.
I am using the same step to retrieve initial token from source node and then to verify on target node.
$ nodetool ring | grep ip_address_of_node | awk '{print $NF ","}' | xargs
Edit: Adding more investigation details ...
I clear the system table data from the node and restart cassandra. But cassandra still picks auto-generated tokens and not those initial_tokens mentioned in cassandra.yml. This clearly violates step 4 of procedure mentioned in official documentation.
So I can rephrase my question as follows: System tables are removed. It's not picking from cassandra.yml. Then where from cassandra picks intial_tokens after restart?
