Data mismatch when querying with different indexes

Question

I stumbled upon with a very curious case. We have a SQL Server 2012 database and such a table

CREATE TABLE [dbo].[ActiveTransactions]
(
    [Id] [BIGINT] IDENTITY(1,1) NOT NULL,
    [Amount] [DECIMAL](12, 4) NOT NULL,
    [TypeId] [SMALLINT] NOT NULL,
    [GameProviderId] [SMALLINT] NULL,
    [UserId] [INT] NOT NULL,
    [Checksum] [NVARCHAR](150) NOT NULL,
    [Date] [DATETIME2](7) NOT NULL,
    [ExternalKey] [VARCHAR](60) NULL,
    [ExternalDescription] [NVARCHAR](1000) NULL,
    [OperatorId] [SMALLINT] NULL,
    [GameId] [NVARCHAR](50) NULL
)

This table has multiple indexes but the two which I want to talk about here are PK_ActiveTransactions (primary key, clustered) which goes:

ALTER TABLE [dbo].[ActiveTransactions] 
    ADD CONSTRAINT [PK_ActiveTransactions] 
    PRIMARY KEY CLUSTERED ([Id] DESC)

And IX_ActiveTransactions_UserIdAmount (nonclustered, non unique):

CREATE NONCLUSTERED INDEX [IX_ActiveTransactions_UserIdAmount] 
ON [dbo].[ActiveTransactions] ([UserId] ASC, [Id] DESC)
INCLUDE ([Amount])

There is a query on which depend major parts of my solution and is called when a certain process is started. Basically every time SomeMethod is called at my code side, it starts SQL transaction, then executes the procedure (shown below), thus locking the entry it selects, then calculates some stuff and inserts new row(s) in that table and commits the transaction. Locking procedure executes this SQL statement

SELECT TOP 1
        id ,
        Amount ,
        TypeId ,
        GameProviderId ,
        UserId ,
        [Checksum] ,
        [Date] ,
        ExternalKey
FROM    ActiveTransactions WITH ( UPDLOCK )
WHERE   @UserId = UserId
ORDER BY Id DESC

Now here is the case. When I was looking over some entries in this table it seemed so that there were multiple (requested at the same time) entries which had selected the same entry for the same @UserId. To be exact, there were 5 new entries (requested at the time, as they were having the same exact [Date] value which is calculated at code side) which all selected the same entry then recalculated some stuff (all 5 of them calculated the same thing) and inserted 5 new rows at the same time, instead of doing that one by one (which should be caused by WITH(UPDLOCK) statement at the end of the SELECT query, as I believe).

Then I tried doing such a thing, I opened three new query windows, I started a transaction with BEGIN TRAN command in one window and then executed above SELECT statement, in other two windows I did the same, When I committed the first statement the second query acquired it right after that, after committing the second statement third one acquired it. (Everything worked as expected), things started looking weird after adding WITH INDEX(INDEX_NAME) (UPDLOCK was still there) at the end of the query. To the first select I specified WITH INDEX(PK_ActiveTransactions) (primary key) and to the other two I specified WITH INDEX(IX_ActiveTransactions_UserIdAmount). After running all 3 of those commands, plus the INSERT in the same table with the first command,(second and third were still waiting for the first to be completed) but when I commited first command, the second acquired the old entry and the third acquired the new entry at the same time. I think that this behavior could have caused the bug explained above, but how is this possible ?

Would SQL Server use two different execution plans (thus using different indexes) for the same query at the same time? This table reaches somewhere 10-15 millions entries at the end of the day, but every morning approximately at 6 AM the jobs executes which leaves the table with 1-2 million rows only. could this cause SQL Server to switch indexes unexpectedly? But anyways I think that this is a series issue meaning that even after committing, indexes may not contain the committed data in them.

The above problem has happened just a few times, I was able to identify them happening twice

Answer 1

You need to inspect what locks are being acquired on your table AND indexes (see link below). SQL Server is able to take separate locks on indexes as well as data. It does not lock all indexes by default.
Note: below is a guess.
Query #1 never acquires a lock on IX_ActiveTransactions_UserIdAmount, therefore query #2 is able to search the index and grab a lock on it and then wait for a row data lock to be released to complete its operation. Once this lock is released query #2 grabs it and holds it, while executing your other code.
Query #3, meantime, is still waiting for both data and index locks. Once query #2 is finished and all locks are released, ONLY then is query #3 able to use the index to do its search and therefore search up-to-date data.

In Summary:
Both query #1 and query #2 are able to search the table in parallel and return the same row. Query #2 does have to wait for query #1 to finish to get the update lock. Since query #1 does not actually modify the last row but rather insert a new one the index is not changed for the purposes of query #2.
See https://www.mssqltips.com/sqlservertip/1485/using-sql-server-indexes-to-bypass-locks/ for discussion about the inverse of your problem.

Additional Comments:
I think it would be more reliable and possibly give better performance for your purposes to lock on "Users" table (if such exists) for a specific user ID rather than rely on index locking working correctly.