Add latency estimations to simulator #21

maaspa · 2024-07-10T09:56:27Z

Changes: Extended the simulator so that it can provide a latency estimation for a given kernel. The code computes the total latency, as well as per-instruction latency and the latency of the longest operation for each instruction.
Tests used to verify: The latency estimations were validated by testing the provided examples as well as my own personal test cases written previously (designed to identify and test challenging patterns). To further verify the accuracy of the estimations, the reviewer can try other kernels (or write new ones) and compare with the expected results simulated on HEEPsilon.

JuanSapriza · 2024-07-10T12:18:37Z

@maaspa please add a detailed description including

What were the precise changes that you did
Which tests did you run to check that it works
What are the things the reviewer (me in this case) should test
If there are any other pending tasks (in which case the PR should be converted to a draft )

JuanSapriza

Nice job. Check the comments, which are mostly naming and placing. I think you approached really fast the kind of architecture i had in mind, well done!
I hope you are seeing the advantages in clarity and scalability of doing it this way 😄

JuanSapriza · 2024-07-10T12:22:33Z

src/cgra.py

@@ -21,6 +22,19 @@
 dsts    = ['SELF', 'RCL', 'RCR', 'RCT', 'RCB','R0', 'R1', 'R2', 'R3']
 regs    = dsts[-4:]

+operation_latency_mapping = {}


Could we move the latency-related operations to a separate module latency.py?
It is unclear to me when this code is gonna execute (when loading the module probably). I would rather have a latency_load_characterization( filename ) function, so we could actually have more than one possible characterization (for example, for the case where you wanna test different scenarios.

Done, I added a function that can eventually be reused for other characterizations

JuanSapriza · 2024-07-10T12:24:13Z

src/cgra.py

@@ -71,6 +85,8 @@ def __init__( self, kernel, memory, read_addrs, write_addrs):
        self.memory     = memory
        self.instr2exec = 0
        self.cycles     = 0
+        self.instr_time = []


can we add units to all variables that refer to a magnitude?
e.g. if instr_time is in seconds, rename it to instr_time_s

if instead it is in clock cycles, instr_time_cc, in which case might be better to call it instr_latency_cc

Done ✔️

JuanSapriza · 2024-07-10T12:25:52Z

src/cgra.py

@@ -97,23 +113,42 @@ def step( self, prs="ROUT" ):
            for c in range(N_COLS):
                self.cells[r][c].update()
        instr2exec = self.instr2exec
+        self.max_instr = None
+        self.lw_count = [0] * N_COLS


self.lw_count was never defined right?

It was part of my previous memory latency algo which I have now removed. It is no longer in my code

JuanSapriza · 2024-07-10T12:33:29Z

src/cgra.py

+                if  self.max_instr is None or self.cells[r][c].time > self.max_instr.time:
+                    self.max_instr = self.cells[r][c]
+                if self.cells[r][c].op in ["LWD", "LWI", "SWD","SWI"]:      
+                    self.lw_count[c] += 1


I dont think this belongs here. This section of the code is controlling the operation of the instructions, and in the middle we are adding a section to count the number of memory accesses. Does not feel right, true?
Probably there is a better place to do it. For example, given that we are saving all the history of operations, computing the number of memory access can very easily be done at the very end by simply counting how many times the operations appear in the instructions matrix.

Regarding the max_instr i find it counter-intuitive. I understand that you are trying to check which cell of this instruction is the one that has the longest operation (therefore max_instr is to be read as maximum latency of this instruction). Had think quite a bit to see it this way. Additionally, given that several cells are doing LW, then why would one cell be the longest or not? It's weird to think and sounds kinda arbitrary. Will keep reading the code to see how it's used, but have a negative feeling about it.

Regarding the ["LWI", ...], please define the array as a global variable with a name as OPERATIONS_MEMORY_ACCESS.

Done ✔️ (moved the code to a more appropriate place)

JuanSapriza · 2024-07-10T12:36:40Z

src/cgra.py

        if PRINT_OUTS: print("Instr = ", self.cycles, "(",instr2exec,")")
        for r in range(N_ROWS):
            for c in range(N_COLS):
                op =  self.instrs[instr2exec].ops[r][c]
                b ,e = self.cells[r][c].exec( op )
                if b != 0: self.instr2exec = b - 1 #To avoid more logic afterwards
                if e != 0: self.exit = True
+                if  self.max_instr is None or self.cells[r][c].time > self.max_instr.time:


When is the time of a cell computed and assigned? Could not find that part. Also, is it time or latency (the word time usually refers to a moment in time and not a period of time). Also add units latency_cc.

Fixed units and added line that fetches latency ✔️

JuanSapriza · 2024-07-10T12:45:14Z

src/cgra.py

@@ -375,6 +411,17 @@ def blt( self,  val1, val2, branch ):
    ops_jump    = { 'JUMP'      : '' }
    ops_exit    = { 'EXIT'      : '' }

+def display_characterization(cgra):
+    total_time = 0


remember to also add units to all variables

Done ✔️

JuanSapriza · 2024-07-10T12:45:51Z

src/cgra.py

@@ -375,6 +411,17 @@ def blt( self,  val1, val2, branch ):
    ops_jump    = { 'JUMP'      : '' }
    ops_exit    = { 'EXIT'      : '' }

+def display_characterization(cgra):


move to another module. We can have one for latency and one for energy (in a year another for area, etc..)

Done ✔️

JuanSapriza · 2024-07-10T12:47:38Z

src/cgra.py

+            print("Cycle:", index + 1, "( ", item.instr2exec, " )")
+            print("Instruction:", item.instr)
+            print("Time:", item.time, "CC\n")
+            total_time += item.time


I would say this information should already be available in the class instance e.g. in self.total_latency_cc

Maybe even better in a self.latency.instructions with a matrix and self.latency.total, self.latency.bottleneck, etc...

Done ✔️

JuanSapriza · 2024-07-10T12:49:44Z

src/cgra.py

+    total_time = 0
+    print("Longest instructions per cycle:\n")
+    for index, item in enumerate(cgra.instr_time):
+            print("Cycle:", index + 1, "( ", item.instr2exec, " )")


For printing information i always recommend a transposition of this. e.g.
Instead of having:

Name: Juan Last Name: Sapriza Age: 22 (?)

Have

Name Last Name Age (years) Juan Sapriza 22 Maxime Aspros 20

I guess you see why 😉
It's also much comfortable to copy-paste to a spreadsheet, or export into a file to open in python later.

Done ✔️

JuanSapriza · 2024-07-10T12:49:58Z

src/operation_characterization.csv

@@ -0,0 +1,27 @@
+# operation_latency_mapping


Maybe you wanna add an extra row with "non-operation-codes", for example, the overhead for memory accesses, and the additional cost of each memory access, the overhead for the first iteration of the kernel, of moving memory, etc...

Done ✔️

JuanSapriza · 2024-07-15T07:10:21Z

src/characterization.py

+        mem_latency_cc += 1
+    self.max_latency_instr.latency_cc = max(self.max_latency_instr.latency_cc, mem_latency_cc)
+    if (self.exit):
+        if (self.max_latency_instr.latency_cc > 2):


Why this tho?

-The "max" function is used to only retain the largest operation (ie: between 3CC multiplication and only one 2CC memory operation, conserve the SMUL)
-An instruction takes an extra CC if it contains the EXIT operation (= last instruction)

JuanSapriza · 2024-07-15T07:12:17Z

src/characterization.py

+    self.max_latency_instr.instr2exec = self.instr2exec    
+    self.instr_latency_cc.append(copy.copy(self.max_latency_instr))
+    self.total_latency_cc += self.instr_latency_cc[-1].latency_cc


What's all this?

First line: we need the current CGRA-cycle number as well as that of the instruction being executed (instr2exec)
Second line: Self.instr_latency is an array containing the longest operation for each instruction
Third line: With each instruction, we also sum up the total latency (to avoid doing so in the display_characterization function)

JuanSapriza · 2024-07-15T07:16:03Z

src/characterization.py

+    self.total_latency_cc += self.instr_latency_cc[-1].latency_cc
+
+def display_characterization(cgra):
+    print("Longest instructions per cycle:\n")


with "cycle" you mean "CGRA-cycle" (i.e. the execution of an instruction)?
If an execution has 1256655 CGRA-cycles, we are gonna print them all?
Why not make use of the system that is already implemented where the user can choose what to print? For example, I could call

run(kernel_name, version=version, pr=["ROUT","OPS", "OP_MAX_LAT", "TOTAL_LAT" ], load_addrs=load_addrs, store_addrs=store_addrs)

Instead of forcing the user to see all the time everything

Good idea, I'm going to do this

JuanSapriza · 2024-07-15T07:33:49Z

src/characterization.py

+            if self.max_latency_instr is None or self.cells[r][c].latency_cc > self.max_latency_instr.latency_cc:
+                self.max_latency_instr = self.cells[r][c]
+            if self.cells[r][c].op in OPERATIONS_MEMORY_ACCESS:      
+                mem_latency_cc += 1
+    # A memory access to a memory bank has a 2-cycle overhead, 
+    # plus 1 additional cycle per PE trying to access it.
+    if mem_latency_cc >= 1:
+        mem_latency_cc += 1
+    self.max_latency_instr.latency_cc = max(self.max_latency_instr.latency_cc, mem_latency_cc)


I find it a little unhappy that this logic has to be hardcoded. This is easy now, but when we speak about the INTERLEAVED BUS you will not want to change the code to switch from one to other. What about this other approach:

operation_characterization_N2M.csv

TYPE, OPERATION, LATENCY, OVERHEAD, ADD_SAME_ROW, ADD_SAME_COL, ADD_CGRA, ADD_ADDR_SAME_INTEGER, ADD_ADDR_SAME_MODULO 0, NOP, 1, 0, 0, 0, 0, 0, 0 1, SADD, 1, 0, 0, 0, 0, 0, 0 1, SSUB, 1, 0, 0, 0, 0, 0, 0 2, SMUL, 3, 0, 0, 0, 0, 0, 0 3, LWD, 0, 2, 0, 1, 0, 1, 0

operation_characterization_ONE2M.csv

TYPE, OPERATION, LATENCY, OVERHEAD, ADD_SAME_ROW, ADD_SAME_COL, ADD_CGRA, ADD_ADDR_SAME_INTEGER, ADD_ADDR_SAME_MODULO 0, NOP, 1, 0, 0, 0, 0, 0, 0 1, SADD, 1, 0, 0, 0, 0, 0, 0 1, SSUB, 1, 0, 0, 0, 0, 0, 0 2, SMUL, 3, 0, 0, 0, 0, 0, 0 3, LWD, 0, 2, 0, 1, 1, 0, 0

Here you kind of embed the logic in the CSV and simplify the logic in the code, so if you want to test different architectures you dont need to modify the code. The example from above is the different bus topologies you already encountered (dont take this as is, as I could be wrong)

The idea is that you encode in the different columns different latency patterns and how they are affected by operations of the same type. For example, ADD_SAME_ROW is the cc you add if you encounter another instruction of the same type in the same row, ADD_ADDR_SAME_INTEGER is, you take the target address, perform the integer division with the memory bank size and, if its the same, add 1 cc of penalty, same for the MODULO with the % modulo division (this will be useful for the interleaved memory banks.

I'm not saying DO IT LIKE THIS ... but this kind of spirit that you will not want to change the code every time. It's something one day we will also do for the instruction operations to make the logic independent of the simulator so we can test different architecture changes without modifying the code. BTW.. the passion for not modifying the code is because we would like to have e.g. 20 different architectures and test which is the best.

actually, given how you implemented the fetch of the values, in the same CSV we could have different implementations:

#operation_latency_cc_mapping . . . #operation_memory_interleaved_latency_cc_mapping . . .

JuanSapriza · 2024-07-15T07:35:27Z

src/characterization.py

+
+OPERATIONS_MEMORY_ACCESS = ["LWD", "LWI", "SWD","SWI"]
+
+def load_operation_characterization(operation_mapping, characterization_type):


why take operation mapping as a parameter?

Fixed ✔️

src/cgra.py

maaspa · 2024-08-06T15:06:27Z

Changes:

The simulator now uses the bus type as a parameter to estimate the latency: 1-M, N-M, and interleaved bus types are supported. In the case of the 1-M bus, the conflicts between memory accesses and the CPU fetching code instructions impact latency. Currently, the simulator only estimates the latency due to the CPU polls regularly to check if the CGRA has finished executing.
Simulator also estimates the configuration time before execution, as well as the time between the end of configuration and the start of execution iteration.
User can specify the output information to be printed by using the pr parameter.
Tests used to verify: The latency estimations were validated by testing the provided examples as well as my own personal test cases written previously (designed to identify and test challenging patterns). To further verify the accuracy of the estimations, the reviewer can try other kernels (or write new ones) and compare with the expected results simulated on HEEPsilon.

JuanSapriza

We really improved the first stages of getting the latency. We now need to continue on the same path for the rest of the computation. From ordering the DMA accesses onwards is still seemingly too complex.

I left some extra comment here and there you might also wanna check

JuanSapriza · 2024-08-06T15:38:12Z

src/cgra.py

@@ -20,7 +21,7 @@
 srcs    = ['ZERO', 'SELF', 'RCL', 'RCR', 'RCT', 'RCB',  'R0', 'R1', 'R2', 'R3', 'IMM']
 dsts    = ['SELF', 'RCL', 'RCR', 'RCT', 'RCB','R0', 'R1', 'R2', 'R3']
 regs    = dsts[-4:]
-
+flag_poll_cnt = 0


This sounds pretty dangerous. This variable is being used outside this module, but only used outside. Although it's technically possible, i would strongly dis-advice it. If it's a variable being used in the context of the CGRA, i suggest adding it as an internal variable of the class and you can initialize it to 0 in the init function.

Also, please leave an extra enter between these lines and the rest. It's nice not to cramp the code to improve readability. The same happens with indenting consequent lines that are doing similar things to the same level (i.e. you align all the = equal signs)

fixed ✔️

JuanSapriza · 2024-08-06T15:41:13Z

src/cgra.py

@@ -47,8 +48,8 @@ def print_out( prs, outs, insts, ops, reg ):
            elif    pr == "R1"   : pnt = reg[1]
            elif    pr == "R2"   : pnt = reg[2]
            elif    pr == "R3"   : pnt = reg[3]
-
-            out_string += "["
+            if pnt != []:


and if pnt == []? The for should still be executed? What was the idea behind it?

JuanSapriza · 2024-08-06T15:46:13Z

src/characterization.py

+    mem_latency_cc = adjust_latency_for_bus(self, mem_latency_cc, bus_type)
+    if mem_latency_cc > self.max_latency_instr.latency_cc:
+        self.max_latency_instr.latency_cc = mem_latency_cc
+        self.max_latency_instr.instr = f'MEM ({self.max_latency_instr.instr})'  


JuanSapriza · 2024-08-06T15:47:34Z

src/characterization.py

+def load_operation_characterization(characterization_type):
+    operation_mapping = {}
+    script_dir = os.path.dirname(os.path.abspath(__file__))
+    csv_file_path = os.path.join(script_dir, 'operation_characterization.csv')


Could we leave the characterization file as a parameter?

JuanSapriza · 2024-08-06T15:49:33Z

src/characterization.py

+                continue 
+    return operation_mapping
+
+operation_latency_mapping = load_operation_characterization("latency_cc")


When is this code executed? When the module is included? Shouldnt this be in a init_characterization or load_characterization or something? Then we should call these functions from the notebook. I haven't finished reading the code, but i find it weird that all this is working without generating any changes on the notebooks launching the examples right?

JuanSapriza · 2024-08-08T08:07:25Z

src/characterization.py

+def compute_bank_index(self, r, c) :
+    if self.bus_type == "INTERLEAVED":
+        if self.cells[r][c].op == "SWD":
+            index_pos = int(((self.cells[r][c].addr - self.init_store[0]) / 4) % 8)


why 4 and 8? What are those numbers?

forgot to change those to constants thanks for the reminder ✔️

JuanSapriza · 2024-08-08T08:10:52Z

src/characterization.py

+                    if self.cells[k][c].op in OPERATIONS_MEMORY_ACCESS and (k, c) not in covered_accesses:
+                        covered_accesses, concurrent_accesses = update_accesses(covered_accesses, concurrent_accesses, r, c, k, self.cells[k][c].bank_index)
+                        break
+    if self.bus_type != "INTERLEAVED":


Why does it matter if the memory is interleaved in the reordering of the DMA? The DMA does not know which memory architecture it will be targeting, so it seems weird that it affects something on how it orders operations.

moved to a more appropriate location ✔️

JuanSapriza · 2024-08-08T08:15:27Z

src/characterization.py

+    concurrent_accesses = [{} for _ in range(4)]
+    # reorder memory accesses to group into concurrent executions
+    # covered_accesses tracks the accesses that have already been visited
+    for r in range(N_ROWS):


So... when you explained me this logic you were just grabbing the operations and "moving them up"... This seems like just creating an NxN matrix and copying them "flattened"... why is such a complex logic required to this? I think these 4 functions could be a single one with 3 or 4 lines

JuanSapriza · 2024-08-08T08:20:04Z

src/characterization.py

+    longest_alu_op_latency_cc = get_latency_alu_cc(self)
+    total_mem_latency_cc = get_latency_mem_cc(self)
+    self.max_latency_instr.latency_cc = max(longest_alu_op_latency_cc, total_mem_latency_cc)
+    if total_mem_latency_cc > longest_alu_op_latency_cc:


and if not?

This if condition only serves to add "MEM" to the longest operation's name when the mem ops take longer than the non-mem ones. If this is not the case, then the name is just that of the longest non-mem operation.

JuanSapriza · 2024-08-08T08:24:19Z

src/characterization.py

+bus_type_active_row_coef = load_operation_characterization("active_row_coef")
+bus_type_cpu_loop_instrs = load_operation_characterization("cpu_loop_instrs")
+
+def get_latency_cc(self):


What is self? the cgra? the instruction? the operation? the cell?

Note that self is a reserved parameter for the parent class instance. That is...

class Person: def __init__( self, height ): self.height = 0 def get_height(self): return self.height juan = Person( 180 ) height = juan.get_height()

self is a parameter YOU DO NOT PASS. It is passed automatically because its the instance of the containing method. Under no circumstances use self as a parameter name.

Also because then you dont know what the parameter is... like was my case

Corrected self to "cgra" ✔️

JuanSapriza

I really like the progress! Well done.
I left some more comments. I know you are now fixing some things with the CPU that were not working.

Please, check this review comments and the previous ones, fix that problem with the CPU accesses and then I can proceed to doing tests to merge :)

JuanSapriza · 2024-08-19T12:03:47Z

src/characterization.py

+                cgra.cells[r][c].bank_index = compute_bank_index(cgra,r,c)
+
+def compute_bank_index(cgra, r, c):
+    base_addr = cgra.init_store[0] if cgra.cells[r][c].op == "SWD" else sorted(cgra.memory)[0][0]  


I dont fully understand what this base address stands for, or why it's different for the SWD...

base_addr corresponds to the address of the first memory element (cgra_input[0][0]). We need this value to find the position of the elements and ultimately return the corresponding bank index. (ie: 9th element accessed -> bank 0).
We need to handle SWD separately due to the starting address.

but why is the starting address related to the bank number?

JuanSapriza · 2024-08-19T12:04:47Z

src/characterization.py

+    if cgra.memory_manager.bus_type == "INTERLEAVED":
+        index_pos = int(((cgra.cells[r][c].addr - base_addr) / cgra.memory_manager.spacing) % cgra.memory_manager.n_banks)
+    else:
+        index_pos = cgra.cells[r][c].addr / cgra.memory_manager.bank_size


shouldnt the one-to-M be another case where the index is always the same (to simulate that they always get a conflict)?

JuanSapriza · 2024-08-19T12:05:54Z

src/characterization.py

+def compute_bank_index(cgra, r, c):
+    base_addr = cgra.init_store[0] if cgra.cells[r][c].op == "SWD" else sorted(cgra.memory)[0][0]  
+    if cgra.memory_manager.bus_type == "INTERLEAVED":
+        index_pos = int(((cgra.cells[r][c].addr - base_addr) / cgra.memory_manager.spacing) % cgra.memory_manager.n_banks)


What is the spacing? if its how many bytes for each address, you may wanna call it alignment_B or word_size_B (my preferred)

Add units to variables so that we know if they are in bits (b) , Bytes (B) or words of 4 bytes (w). My standard is to add it as a underscore and then the unit word_size_B, banks_n (with n standing for something that is just a count and therefore has no units)

JuanSapriza · 2024-08-19T12:10:16Z

src/characterization.py

+                return pairs[1].index(pair)
+    return 0
+
+def compute_latency_cc(cgra, dependencies):


maybe wanna give it a name that refers to memory? because its not latency in general, but the latency of a memory access, i understand

JuanSapriza · 2024-08-19T12:10:56Z

src/memory.py

+WORD_SIZE   = 4
+
+class MEMORY:
+    def __init__( self,bus_type="ONE-TO-M", spacing=4, n_banks=8, bank_size=32000):


nice. Just add the units

JuanSapriza · 2024-08-19T12:11:59Z

src/memory.py

+        self.bank_size = bank_size
+        self.flag_poll_cnt = 0
+
+def kernel_clear_memory( name, version=""):


if you want, you can remove the work kernel from the beginning of these functions, as it does not make sense anymore. I like it more starting with memory as they are in the memory module.

estimator can simulate latency

fa4016d

JuanSapriza requested changes Jul 10, 2024

View reviewed changes

maaspa added 2 commits July 11, 2024 12:14

added characterization.py

8f403b8

removed comments

3141ac5

maaspa requested a review from JuanSapriza July 12, 2024 10:34

JuanSapriza requested changes Jul 15, 2024

View reviewed changes

estimator uses bus type to simulate latency

fe3d72d

divided latency func into steps

22cc30c

JuanSapriza requested changes Aug 8, 2024

View reviewed changes

maaspa added 2 commits August 8, 2024 16:32

refactored latency algo

b46f6d2

slight refactoring

770f2da

JuanSapriza requested changes Aug 19, 2024

View reviewed changes

maaspa and others added 2 commits August 26, 2024 10:34

renamed var names / fixed cpu estimation logic

de70c2c

test commit

4b563b8


		OPERATIONS_MEMORY_ACCESS = ["LWD", "LWI", "SWD","SWI"]

		def load_operation_characterization(operation_mapping, characterization_type):

Add latency estimations to simulator #21

Are you sure you want to change the base?

Add latency estimations to simulator #21

Conversation

maaspa commented Jul 10, 2024 • edited Loading

JuanSapriza commented Jul 10, 2024 • edited Loading

JuanSapriza left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JuanSapriza Jul 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maaspa commented Aug 6, 2024

JuanSapriza left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JuanSapriza left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maaspa commented Jul 10, 2024 •

edited

Loading

JuanSapriza commented Jul 10, 2024 •

edited

Loading

JuanSapriza Jul 10, 2024 •

edited

Loading