A method of processing database queries according to an embodiment of the disclosure includes the steps of acquiring an input query of a database, generating a join graph consisting of vertexes corresponding to the tables of the database and edges corresponding to join information among the tables, dividing the join graph into a plurality of subgraphs, and for the input query, generating an execution plan of a target query including N-ary join operators by using the plurality of subgraphs, optimizing the execution plan of the target query based on a cost model, acquiring a table sort algorithm of the N-ary join operators included in the target query based on an available size of a GPU or CPU memory, determining a sort order of one or more join columns included in the tables of the N-ary join operators, and based on the sort order of the one or more join columns, performing an operation for the plurality of subgraphs.