You aren't escaping your periods. Try like this:

location ~* \/(fileA|fileB)\.txt$ {
        return 404 'nothing here';
}

~~Never mind, I understood my mistake... This nginx built didn't come with ngx_http_rewrite_module so return will not works.~~

You might have another location block before it that catches the route.

Actually I don't think the error lay in the regex.

because

	location /fileA.txt {
		return 404 'nothing here';
	}

Does not works either...

Finally it works as it should I update my old nginx version to newest freenginx and read Thoroughly the location documentation and this post thanks to @ramble81@lemmy.zip